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Foreword 



This publication is a very significant cooperative 
effort of the Department of Audiovisual Instruction and 
the National Society for Programmed Instruction. It is, 
we believe, a harbinger of future joint activities between 
our two organizations whose purposes converge in the 
field of programmed learning. 

Since 1923 the Department of Audiovisual Instruc- 
tion of the National Education Association has been 
a major source of information to the teaching profes- 
sion on what has recently come to be known as 
educational technology. Through its magazine. Audio- 
visual Instruction, its quarterly journal, AV Communi- 
cation Review, and its dozens of other publications, 
DA VI and its 6,000 members have worked diligently 
to improve instruction through the more effective use 
of all educational media in the teaching-learning 
process. More recently the National Society for Pro- 
grammed Instruction, described in the Preface, has 
become a significant source of information on pro- 
grammed learning with its journal and its convention. 

It seems most appropriate that DAVE and NSPI 
have joined together to produce this important volume 
for their own members, for the teaching profession. 



and for all those interested in this promising new 
instructional technology. 

On behalf of the members of both DAVI and NSPI, 
we want to take this opportunity to express our grati- 
tude to the editors of this volume. Both Col. Ofiesh, 
editor, and Dr. Meierhenry, associate editor, have 
worked tirelessly with both insight and dedication to 
make this book a reality. Members of the NSPI Edi- 
torial Board have each made important contributions 
and we want to express our thanks to each of them: 
David M. Wark, William A. Deterline, Carlton B. 
Downing, M. C. Zaccaria, Charles F. Adams, A. J. 
Bouck, I. J. Newman, and John R. Olsen. 

Finally, we extend thanks to those who wrote this 
book, the distinguished group of authors and pioneers, 
whose papers we are pleased to publish. 

WILLIAM H. ALLEN, President 
Department of Audiovisual Instruction, 

National Education Association 

ROBERT G. SMITH, Jr., President 
National Society for Programmed Instruction 
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Preface 



In the first week of January, 1962, eight individuals 
interested in programmed instruction met. Although 
most of these individuals were given to great flights 
of imagination and expectations, none expected that 
a National Society for Programmed Instruction would 
be formed as a result of their activities. Certainly none 
expected that such a national society would hold its first 
convention in San Antonio participated in by the most 
prominent names in the field of programmed instruc- 
tion, and which has resulted in the preparation of this 
present volume, just a little more than a year and a 
half after this original meeting. 

The original eight along with thirty-two other indi- 
viduals met in February, 1962, at Randolph AFB, 
Texas, to found the National Society for Programmed 
Instruction whose purpose was to “collect, develop and 
diffuse information concerned with programmed in- 
struction.” The original group represented a cross- 
section of San Antonio Military, Public School and 
University education and training officials. The group 
elected Colonel Gabriel D. Ofiesh, USAF, as its Presi- 
dent, Dr. Carlton B. Downing, Trinity University, Vice 
President, Colonel Marian A. Tierney, USA, Secretary, 
and Dr. Michael A. Zaccaria, Psychologist, Treasurer. 

At this first meeting the President spoke of the re- 
sistance meeting the introduction of programmed in- 
struction. This resistance he felt was largely the result 
of ignorance and misinformation about programmed 
instruction. He charged the members of the society 
with the responsibility to spread and inform the public 
about the potentialities of programmed instruction and 
to dispel both ignorance and misunderstanding. 

On April 14, 1962, the National Society for Program- 
med Instruction (NSPI) presented the first “Program- 
med Instruction Institute” in San Antonio, Texas. The 
institute, held at Trinity University, was presented to 
introduce the “why,” “what,” and ' how” of program- 
med instruction. The audience was a heterogeneous 
one, ranging from sophisticated research psychologists 
to insurance salesmen curious about the “teaching 
machine” business. The speakers and discussion lead- 
ers tried, and admirably succeeded, in presenting some- 
thing for everyone. 

In the first paper of this institute, NSPFs president 
pointed out three nation-wide problems of modem edu- 
cation: high-school dropouts, industrial retraining, and 
teacher procurement. He suggested that programmed 



instruction may be the first step toward a partial solu- 
tion to some of these problems. Cautiously, however, 
he put the new techniques into perspective, balancing 
the proposed solutions against some of the adverse 
effects they may have on education, tr ainin g and our 
total society. 

The papers contained in this volume are, among 
many things, a record of those who labored, unceasingly 
and without formal recognition, to make NSPI a truly 
national society. We should like to express our sincere 
appreciation to the author's of the papers for granting 
permission to make slight editorial revisions as well as 
condensations. 

A special word of appreciation is due the Department 
of Audiovisual Instraction (DAVE) of the National 
Education Association. We wish to express our sin- 
cere appreciation for the whole-hearted support and 
encouragement provided by the entire DA VI staff. 
Specifically to be commended in this respect is Dr. 
Robert C. Snider, Assistant Executive Secretary of 
DA VI. From the very beginning of this venture, he has 
encouraged the idea of a joint publication of these pro- 
ceedings. Throughout the editorial phases of the pro- 
duction of this book, he provided us with constant 
guidance and assistance. Without his moral and emo- 
tional catalytic support, it is doubtful that this publica- 
tion would have become a reality. Lastly, the editors 
wish to express their deep and sincere debt to Lt. 
Charles F. Adams and Mrs. Peggy Langner, for their 
untiring efforts over the entire period of the convention, 
and who, during preparation of this volume, worked 
many long hours and supervised the work of others in 
the preparation of the intermediate drafts of all the 
papers. It is our hope that this v/ill be only the first of 
a series of volumes which in the years to come will 
define future trends and innovations in instructional 
and educational technology. 

GABRIEL D. OFIESH, Ed.D. 

Colonel, USAF 

Director, Training Methods Directorate 
Hq ATC, Randolph AFB, Texas 

WESLEY C. MEIERHENRY, Ph.D. 

Assistant Dean 
Teachers College 
The University of Nebraska 
Lincoln, Nebraska 
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THE CARPENTRY REVOLUTION— A FABLE* 
W. A. Deterline 



Once upon a time carpentiy was a very frustrating 
business which depended on tools that rarely produced 
products of consistent quality. Carpenters labored 
hard with what they had. but their products were ac- 
cepted only because there seemed to be no possibil- 
ities of doing any better. Then one day a man who 
was not even a carpenter invented a tool he called a 
“plane” and he suggested that carpenters could use 
it to great advantage in their work. No one listened to 
this man, of course. Some years later another who 
was not a carpenter either, invented a device he called 
a “saw” and he made quite a bit of noise about his de- 
vice and its potential in the carpentiy business. Another 
man (also non-union) at about the same time invented 
a “hammer” and he too made noise that attracted 
attention. Each of the three men attracted followers, 
some of whom were cooks and hunters and golf pros. 
Then a peculiar thing happened. Instead of providing 
a useful set of tools for the carpenters, the “saws” 
and the “planes” and the “hammers” argued among 
themselves over which tool was best. The “saws” 
said, “Our tool works on wood better than a hammer 
or a plane,” and the hammers said, “Our tool hits nails 
and thumbs better than a saw or a plane”, and the 
planes, well you know what they said. Then other 
voices were heard extolling the virtues of screw driv- 
ers, rasps, drills and all manner of other strange tools 



and furious controversies were waged over the question 
of who had the one tool that would solve all of the 
carpenier’.s problems. A few unaware types tried using 
combinations of several tools, but they were quickly 
hooted cut of town for their lack of imagination. The 
carpenters waited and waited for the dust to settle so 
they could welcome the winner with a ticker-tape pa- 
rade, but lost patience as the bickering continued and 
finally just shrugged their shoulders and went back to 
work. Since everybody in the new tool missionary pro- 
fession was so involved in his own little sphere of in- 
terest, the whole business finally collapsed. The cooks 
went back to cooking; the hunters went back to hunt- 
ing; silence feel in the land. Several centuries later 
someone stumbled onto the records of the old contro- 
versies. After laughing at the silliness and futility of 
it all, he put together a highly flexible and effective 
tool kit that made carpentry the science it is today. 
Then many people lived happily ever after some of the 
time. 

MORAL: Our classrooms could have had comfortable 
seats sooner if someone had kept his eyes on the end 
not the means. 



*The fable as a communications device was first used ob- 
jectively by the Aesop of programmed instruction. Dr. Robert 
F. Mager. 



I Introduction 

Gabriel D. Ofiesh, Colonel, USAF 



As we nm down the list of education and training 
problems confronting us today, we might well feel we 
were marooned on an island of problems surrounded 
by a sea of despair. But when I look at this sea of 
despair surrounding this island, I am reminded of the 
story that I read a few days ago, which is attributed 
to Gilbert Keith Chesterton and which may be apocry- 
phal. Chesterton was asked one day to answer the pro- 
found question that, if he were marooned on an island 
with no hope for survival, what is the one thing he 
would want above aU things, if he had his choice. He 
answered very practically, “I think I would ask for a 
copy of Jame’s text on Practical Ship Building.” Now, 
being as deeply a religious man as he was, it is interest- 
ing that he didn’t ask for a message, or a panacea. He 
didn’t ask for the Bible or for a miracle. Self-confident 
in his relationship to his God, he simply asked for a 
means to survival. 

This anecdote illustrates the need for realistic inno- 
vation in most trying circumstances. We are on the 
threshold of radical innovation in the field of education. 
Programmed instruction has emerged as simply a vehi- 
cle througji which this educational innovation may 
make substantial strides. John Gardner, President of 
the Carnegie Corporation, in his Annual Report last 
year (Gardner, 1962) had the following comments to 
make about innovations which are pertinent to those of 
us who would innovate in education in our society 
today: 

To accomplish renewal, we need to understand what pre- 
vents it. When we talk about revitalizing a society, we tend 
to put exclusive emphasis on finding new ideas. But there 
is usually no shortage of new ideas; the problem is to get a 
hearing for them. And that means breaking through the 
crusty rigidity and stubborn complacency of the status quo. 

The development of resistance to new ideas is a familiar 
process in the individual. The infant is a model of openness 
to new experience, receptive, curious, eager, unafraid, willing 
to try anything. As the years pass these priceless qualities 
fade. He becomes more cautious, less eager, and accumulates 
deeply rooted habits and fixed attitudes. 

The same process may be observed in organizations. The 
young organization is willing to experiment with a variety of 
ways to solve its problems. It is not bowed by the w'eight 
of tradition. It rushes in where angels fear to tread. As it 
matures it develops settled policies and habitual modes of solv- 
ing problems. In doing so it becomes more efficient, but also 
less flexible, less willing to look freshly at each day’s experi- 
ence. Its increasingly fixed routines and practices are congealed 
in an elaborate body of written rules. In the final stage of 
organizational senility there is a lule or precedent for every- 
thing. Someone has said that the last act of a dying 



organization is to get out a new and enlarged edition of the 
rule book. 

And written rules are the least of the problem. In mature 
societies and organizations there grows a choking underbrush 
of customs and precedents. There comes to be an accepted 
way to do everything. Eccentric experimentation and radical 
departures from past practice are ruled out. The more 
pervasive this conventionality, the less likely is the innovator 
to flourish. The inventor of the Bessemer process for steel 
making. Sir Henry Bessemer, wrote: 

T had an immense advantage over many others dealing with 
the problem inasmuch as I had no fixed ideas derived from 
long established practice to control and bias my mind, and 
did not suffer from the general belief that whatever is, is 
right.’ 

As a society becomes more concerned with precedent and 
custom, it comes to care more about how things are done 
and less about whether they are done. The man who wins 
acclaim is not the one who “gets things done” but the one 
who has an ingrained knowledge of the rules and accepted 
practices. Whether he accomplishes anything is less im- 
portant than whether he conducts himself in an “appropriate” 
manner. Thus do men become the prisoners of their 
procedures. 

. . . New developments in a field often originate outside 
the area of respectable practice. The break with traditional 
art was not fostered within the Academy. Jazz did not 
spring from the bosom of the respectable music world. The 
land-grant colleges, possibly the most impressive innovation 
in the history of American higher education, did not spring 
from the inner circle of higher education as it then existed. 
Motels, the most significant development of this generation 
in innkeeping, were at first regarded with scorn by reputable 
hotel people. 

Vested interests constitute another problem for the aging 
society. The phrase “vested interests of workers” may be as 
strong as those of the top executives. In any society many 
established ways of doing things are held in place, not by 
logic nor even by habit, but by the enormous restraining 
force of vested interests. In an organization certain things 
remain unchanged for the simple reason that changing them 
would jeopardize the rights, privileges, and advantages of 
specific individuals, perhaps the president, perhaps the main- 
tenance men. 

The more democratic an organization or a society the more 
clearly it will reflect the interests of its members. So a 
democratic group may be particularly susceptible to the 
rigidifying force of vested interest. 

Still another reason for the loss of vitality and momentum 
in a society is a lowered level of motivation. It is not always 
easy to say why motivation deteriorates. Perhaps people 
stop believing in the things they once believed in, the things 
that gave meaning to their efforts. Perhaps they grow soft 
from easy living. Perhaps they fall into the decadent habit 
of imagining that intense effort is somehow unsophisticated, 
that dedication is naive, that ambition is a bit crude. Or per- 
haps a rule-ridden society has bottled up their energy, or 
channeled it into all the tiny rivulets of conformity. 



Trends In Programmed Instruction 

One may argue, as Toynbee does, that a society needs 
challenge. It is true. But societies differ notably in their 
capacity to see the challenge that exists. No society has ever 
so mastered the environment and itself that no challenge 
remained; but a good many have gone to sleep because they 
failed to understand the challenge that was undeniably there. 

Whatever the reason for loss of motivation, the consequences 
are apt to be devastating. Nothing, neither wealth nor 
technology, neither talent nor wisdom, will save a society in 
which motivation continues to deteriorate. 

Dr. J arnes W. Laurie. President of Trinity University, 
in his welcoming remarks at the first annual meeting 
of the National Society for Programmed Instruction 
also pointed out the need for realistic innovation when 
he said: 

Most of us are aware of the fact that information is 
accumulating at such a fast pace that it becomes mandatory 
that we seek out new methods just to cope with the acquiring 
and dissemination of new knowledge as it crowds upon us in 
almost every field. 

And then later in his remarks he stated: 

. . . Any program that will speed up our task and sharpen 
the best brains of the future, more wisely and rapidly, 
deservts the very careful assessment of everyone who says 
he believes in education. As a matter of fact, I think (those 
attending this convention are) dealing in this area with the 
essence of academic freedom. As an educator, I see that the 
administrator or the professor who talks loudly about academic 
freedom is often one who expresses fearfulness when any new 
technique or new procedure is suggested to him, but clutches 
to himself only those timeworn methods with which he is 
most familiar. It is the essence of academic freedom to be 
unafraid of new truths, and new methods, and new ways. I 
for one, and I think I speak for my colleagues on the hill, for 
the other educators in this great community, am eager and 
ready to listen to anything you have to say to us that will 
enable us to do our job more effectively, more wisely, and 
more completely. "... It should not be required of you to give 
us all the answers; it will be necessary that you give us some. 
... I do not think you need to defend the theory any longer. 

I do believe it is important for you to tell what programmed 
instruction is actually doing, what it’s accomplishing, what 
it’s achieving, and what you propose to have it do. 

After these few challenging keynote comments to the 
conference participants, President Laurie went on to 
point out in rather critical, but realistic and hopeful 
terms, that he had no illusion that programmed in- 
struction was “ever going to replace the professor”. 
However, he did point out that this would be small con- 
solation to some because: 

. . . Now some are going to be replaced simply because 
they have a sheer inability at keeping up with the stream 
of knowledge in their field. Some are going to be replaced 
because they’re not willing to pay the terrific price of assimu- 
lating this new knowledge that comes their way. Some are 
going to be replaced because they are not sufficiently addicted 
to academic freedom to be v/illing to try to understand any 
thing new. But most of them, at least the majority of those 
that I know, will be stimulated and aided by what you’re 



doing and will add to their repertoire of skills and techniques 
and methods some of the fruits of your labors. 

The papers in this volume, which reflect the com- 
ments made by the many participants in the confer- 
ence, are a partial answer to Dr. Laurie’s challenge to 
“tell what programmed instruction is, what program- 
med instruction is actually doing, what it is accom- 
plishing, what it is achieving, and what (we) propose 
to have it do.” It is for this reason that the present 
volume is called “Trends in Programmed Instruction”. 
The papers were organized to reflect trends in a vari- 
ety of areas such as: Education; Development of Sys- 
tems Applications; Impact on the Teacher; The Excep- 
tional Student; Industrial Applications; Military and 
Governmental Applications; TTie Health Sciences; In- 
structional Programming; Evaluations; and Future 
Trends. Finally, Dr. Meierhenry’s chapter “A Point in 
Transition”, weighs the parameters for future lines of 
consideration. 

Even though programmed instruction has not pro- 
duced the startling revolution in education that was 
predicted for it several years ago by many of its 
ardent and enthusiastic adherents, a review of the de- 
velopments at the first conference of the National So- 
ciety for Programmed Instruction certainly leaves no 
question but that programmed instruction has at least 
emerged as a truly formidable system of instructional 
technology. It has raised many questions about in- 
structional methods that have been undoubtedly raised 
before, but which today have hope of being realistically 
answered. 

A need for a revolution in educational processes is 
still a necessity for our “Atomic and Educational Age 
of Anxiety”. Even though the results of programmed 
instruction have not been revolutionary, the efforts 
certainly could not be characterized in the same evo- 
lutionary tones as for other educational developments 
which have been simply evolving for centuries. 
Through programmed instruction we have, at the very 
least, arrived at the threshold of a revolution which in 
the years to come may take some very unanticipated 
forms. The efforts of the present state of program- 
ming are primitive, at best, but they are efforts, 
and unusual ones at that. 

Automated instruction with its correlative technology 
still promises the first true and radical innovation in 
instruction since the invention of movable type. This 
is not to gainsay that there have been vast social de- 
velopments which have had an extensive impact on 
education. Education has been extended to millions 
who would not ordinarily benefit from it. Radio, tele- 
vision, and other mass media of communication and 
propaganda have had their undeniable effects. There 
have been significant advancements in the design of 
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school buildings. Despite these and many other devel- 
opments, the fact still remains that there has been no 
revolutionary breakthrough in the processes of educa- 
tion and training. There has been no radical advances 
in instructional theory, methods, or in the development 
of procedures by which people can learn more rapidly 
and more effectively. 

In spite of the numerous education and training prob- 
lems confronting us in our society today, there has 
been no significant effort to provide hopeful solutions. 
The problems, on the other hand, are too numerous to 
mention. The newspapers reported last year that in this 
year alone almost a million lost, bewildered, hopeless 
youngsters will leave school before graduation and 
enter a world which really has no place for them. Dr. 
Daniel Schrieber, Director of the School Drop-Out 
Project of the National Education Association char- 
acterized these youngsters as “constantly running 
from work half-done, from school half-completed, . . . 
truly fugitives from failure.” 

The problems surrounding the impact of technology 
and automation bombard us from all points of the cul- 
tural compass. We have a state of overemployment in 
our unskilled labor ranks and underemployment in our 
skilled trades. How we are going to solve these prob- 
lems will in all likelihood determine our future as a 
nation. 

Judge Mary Kohler, a member of the late President 
Kennedy’s National Committee on the Employment of 
Youth, has emphasized that in 1960, of the 2,500,000 
youngsters who reached 18, roughly one-third went to 
college, one-third quit school after graduation, and 
one-third had dropped out. More tlian one and one-half 
million of these youngsters did not have an organized 
plan to take them from school to a job, that would be 
satisfying to themselves or that would enable them 
to make any significant contribution to society. Dr. 
Conant has pointed to the “social dynamite” inherent 
in this situation. Throughout the country in the past 
year the demand for jobs on the part of late adoles- 
cents has increased nearly seven times and the supply 
has only doubled. Cities throughout our nation are re- 
porting drastic decreases in jobs available to sixteen 
and seventeen-year-olds. 

There is a national awareness of the retraining prob- 
lems that have been created by the impact of automa- 
tion. The problems concerning retraining have been 
more easily stated than solved. These problems are 
not going to be solved by simply cataloguing the num- 
ber and types of jobs which need to be filled with skilled 
workers and then establishing training programs for 
the unemployables to enable them to fill these jobs. 

In discussing our training problems as a nation. 
Congressman Henry B. Gonzales, who gave the keynote 
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address at the fftst annual conference of NSPI, pointed 
out in rather vivid terms the stake that the country has 
in human resources that are not adequately trained. 
The following excerpts from his keynote remarks stress 
in strong terms the need for educational innovation and 
the hopeful note that programmed instruction offers 
without being necessarily the ultimate answer: 

. . . The Federal Government, which is only a very junior 
partner in the total educational endeavors of this Republic, 
spends something in excess of 1,938,155,985 (almost 2 billion 
dollars!) a year right now for teaching functions. This does 
not count the costs of training its own employees or military 
personnel for most governmental positions, but is simply an 
estimate of what is now budgeted for teaching functions to 
citizens. 

... It is used here to give some measure of why the 
government has a direct interest in the work you are going. 

. . . The Manpower Retraining Program is still a fledgling. 
It is now established public policy that we have a commitment 
to help raise the skills, the literacy, the abilities of the 
American people. We have this commitment to help soften 
the dislocations and impact of this thing called automation; 
we must exert ourselves in this area to lessen the costs and 
debilitating affects of our welfare rolls, our unemployment 
lines, our undeveloped human resources. 

... I can conceive of reaching a saturation point on many 
things prominent in our culture: automobiles, food, houses, 
even clothes. But I cannot even conceive of a saturation point 
on human learning, on educational opportunities, on teachers 
themselves. Other jobs may disappear, other tasks 'may be 
removed, but our commitment to learning must by the very 
nature of our world expand and keep on expanding. But the 
point is obvious: With a limitless need, we acquire a limitless 
outlet for the techniques you (NSPI) are developing. 

. . . This is an exciting thing you propose. I have enough 
wisdom, I believe, to know that it is not the only approach 
to educating, nor is it necessarily the last word or the best 
word even in the area where applied. What is exciting is that 
the teachers themselves are flnding new ways to apply their 
art. 

. . . There is a ferment in educational circles today. There 
is much talk of and about methodology, of tools, of content. 
This is good. This bodes well for our future. I suspect out 
of this ferment will come the means of dealing with our future. 
We should not be dismayed because there is not unanimity on 
the proposals being made. How can there be when the form 
of our future is itself unknown? For this reason, if none other, 
we need to have new techniques applied to maintain the 
highest possible flexibility to meet the many unknown factors 
in the equation of our times which inexorably will balance 
what we get out of life with the efforts we put in. 

There was noted a sense of cautious urgency in the 
Congressman’s remarks. It should be emphasized, 
however, that there is no “crash program” that can 
possibly deliver us from our dilemma. We are already 
a nation with a shortage of 150,000 teachers. Many of 
our teachers are non-college graduates having sub- 
standard credentials. The number of able and master 
teachers among those who have paper qualifications 
and college degrees is woefully low. Teachers, as a 
group, still have a long way to go in developing an 
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extremely high professional status. This is largely due 
to the fact that they lack, at present, a true instructional 
technology. Many of them work under impossible con- 
ditions which prevent them from making most effec- 
tive use of their talents and skills. 

The problem, as it appears at present, will get worse, 
not better. Within another three years alone we will 
increase our present school enrollment by five and 
one-half milli on students. Our teacher shortage will 
grow to over 300,000. To merely maintain the status 
quo, over one-half of all our college graduates during 
the next five years would have to become teachers. 
This is a most unlikely prospect. One of our Midwest- 
ern states anticipates a 45% increase in college enroll- 
ment during the next two years. We are likely to react 
to this problem by stressing quantity over quality and 
by reducing still further our teacher qualifications. 

Ours is an age in which our knowledge of man and 
his universe is doubling every decade. It has been 
pointed that there have been more discoveries in the 
field of mathematics during the last decade than there 
have been in all the previous history of civiliation. Our 
educational technology has not been able to keep up 
with the innovations of technology itself. We must turn 
with hope to the belief that the problems created by 
technology wiU be solve;^ by technology. Can instruc- 
tion truly become a science? Can the science of learn- 
ing be applied to the art of pedagogy? Technology is 
the application of science to art. Can the studies of the 
behavioral scientist concerning the fundamental proc- 
esses of learning be applied to the practice of educa- 
tion and instruction? If instruction must remain an art 
and never become a technology, what is our alterna- 
tive? And ultimately, will our desperate need for an 
instructional technology be sufficient in and of itself to 
force the issue? 

The late President John F. Kennedy, in a recent 
message to Congress (1962) emphasized that: 

No task before our nation is more important than expand- 
ing and improving the educational opportunities of all our 
people. . . . 

He further pointed out in his message: 

. . . education is both the foundation and the unifying 
force of our democratic way of life — it is the mainspring of 
our economic and social progress — it is the highest expression 
of achievement in our society, enabling and enriching human 
life. In short, it is at the same time the most profitable 
investment society can make and the richest reward it can 
confer. Educational failures breed delinquency, despair, and 
dependence. They increase the costs of unemployment and 
public welfare. They cut our potential national economic 
output by billions. They deny the benefits of our society to 
large segments of our people. They undermine our capability 
as a nation to discharge world obligations. All this we 
cannot afford — ^better schools we can afford. 



. . . We must provide facilities for fourb^n million more 
elementary, secondary school, and college students, by 1970, 
an increase of thirty per cent. . . . We must find the means 
of financing a seventy-five per cent increase in the total cost 
of education — another $20 billion a year for expansion and 
improvement — ^particularly in facilities and instruction, which 
must be of the highest quality if our nation is to achieve its 
highest goals. 

Here, again, although all the measures mentioned by 
the late President are undoubtedly necessary, they will 
fall short unless some means is provided for making a 
breakthrough in the processes by which some teach and 
others learn. If an infinitesimally small part of the 
effort recommended were to be expended in developing 
a true educational technology, then the tools might 
. be forthcoming by which we might make a momentous 
step forward in the solving of our problem. 

What has aU this to do with programmed instruction 
and the deliberations of the conferees of the fiorst annual 
convention of the National Society for Programmed 
Instruction? There has been no development in educa- 
tion that has provided as hopeful a sign for the revolu- 
tionary emergence of such an educational technology 
as was started by the efforts of Dr. Sidney Pressey some 
35 years ago and so dramatically revitalized by Dr. 
B. F. Skinner a few years ago at Harvard. It is this 
development, which has been referred to as pro- 
grammed instruction or automated teaching, which re- 
lates tile deliberations of the first annual conference of 
NSPI to the need for educational innovation. 

The interest in programmed instruction has been ac- 
companied by much apprehension, anxiety, and talk 
about automation, automated instruction, and teach- 
ing machines. George Kneller (1961), in his address 
before the American Psychological Association, stated 
the issue very succintly when he commented in his 
opening remarks that: 

Whether we like it or not, automated teaching is here to 
stay. Merely to oppose it is futile. Education must mirror the 
age it strives to improve. It cannot isolate itself from automa- 
tion any more than from other social or economic changes. 
For automated teaching is one more of the applications of 
technology to human life. The question to be asked is not, 
‘Do we accept automation?’, but ‘How much of it and under 
what condition?’. 

What has programmed instruction to do with auto- 
mated teaching, and specifically with teaching ma- 
chines? We have heard over and over again that teach- 
ing machines do not teach. It has been pointed out by 
many that they merely present materials which have 
been developed, or programmed as we now say, in a 
special way. Many have claimed that teaching machines 
are simply a device which presents stimuli and provides 
for active responses from the student. Many teachers 
are often merely stimulus machines. They cannot 



provide for active student response except in a tutorial 
relationship, which is impossible to develop with every 
student in a lock-step situation involving 20 or 30 stu- 
dents. Whether this will remain so in the years to come 
will only be answered by further developments in 
teaching machine technology. The following clarifying 
comment from Dr. Robert Mager (1962) deserves 
mention: 

When one says that Teaching machines do not teach,’ 
the implication apparently is that teachers do teach. The 
problem, I suppose, is with the meaning of the word ‘teach.’ 
If a teaching machine is a thing that presents information to 
the learner and adjusts itself on the basis of the learner’s 
response, then so does the teacher. As a matter of fact, the 
machine can be far more effective than the teacher in pre- 
senting information in an appropriately organized fashion, and 
in accommodating itself to the needs of the individual 
learner. If you mean to imply that the difference is one of 
function, and that the teacher does something more than 
simply present the program, then you might say this rather 
than that machines don’t teach but teachers do. While 
instructional devices that do moie than simply present the 
program do not abound, they do exist and will become con- 
siderably more sophisticated as time goes on. Whether a 
device can or should be able to do an}^hing a teacher does 
is another matter. 

One cannot argue at this time, however, with the con- 
tention that the manner in which the information is 
prepared to go into the machine, the size of the incre- 
ments of the information presented, the sequence of the 
information, and how the student responds to the mate- 
rials throughout the program, determine how effective 
the machine really is. This content and the manner in 
which it is prepared is called “the program”, and for 
this reason “the program” is the heart of any teaching 
machine. The “program”, for that matter, is the heart 
of any teacher — ^human or machine. 

Good teachers have always known that their basic 
function was to communicate effectively with students. 
They have always known that they must determine 
how effective or ineffective their communication 
has been before they modify what they have done. 
Their modification was based on knowing where, how, 
and why they had failed in the communication process. 
They did this by listening to the student. Until the loop 
was closed they were not able to determine whether 
instruction had taken place. Good teachers have al- 
ways known that very few students learn very many 
specific facts from teachers. They have perceived 
their role as motivating students to acquire much 
knowledge from reading, studying, analyzing, and 
cogitating. Good teachers inspire, motivate, and illu- 
minate. They don’t teach facts. They teach students. 
They do this best in one specific kind of student- 
teacher relationship, the tutorial relationship. 
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It is obviously impossible for every teacher to be a 
tutor for every student. We will never be able, no 
matter how fine our intentions, to establish a tutorial 
relationship between a master teacher and every stu- 
dent in every classroom in this nation. Some means 
must be found to mass produce the elements of the tu- 
torial relationship so that very student at his own level 
can have quality instruction. 

Programmed instruction is the first step in an effort 
to do this, an effort to package for all students the 
characteristics of the tutorial approach! 

By programmed instruction the student is led “one 
step at a time” along the learning path. He actively 
responds to a curriculum that has been logically se- 
quenced so that his response always carries him a lit- 
tle closer to the ultimate desired learning outcome. 
The student is required to respond to questions, solve 
problems, or complete exercises. Whenever the student 
makes a response he is immediately informed how cor- 
rect or accurate his answer is and, where necessary, 
provided with additional information to correct his an- 
swer. It is this aspect of programmed instruction which 
comes the closest to approximating the tutorial rela- 
tionship. 

The ideal program, according to Skinner, would, be 
one where the student would make no errors whatso- 
ever in his responses. To accomplish this goal, the in- 
formation is presented to the student in very small 
steps, and he is steadily cued, or prompted, in such a 
manner that he cannot help but make the correct re- 
sponse. Learning becomes almost effortless, but not 
thoughtless, and it is meant to be that way. Every re- 
sponse made by the student is an overt as well as a 
covert one. It is the nature of the constructed response 
on the part of student (i.e., filling in a blank, drawing 
a diagram, solving a problem, writing a word, etc.) 
which determines the extent to which he participates 
actively in the learning program. After he constructs 
the correct response he immediately knows whether or 
not he is correct, and this knowledge, or feedback, is 
supposedly reinforcing. The burden is on the program, 
however, to lead the student to construct the correct 
response in such a manner that he learns by doing so. 

All programmed materials are crucially dependent 
on another prior step which experience has shown is 
probably the most important aspect of this technology. 
This is the “task” or “learning outcome analysis.” 
Traditional descriptions of learning objective have 
been most inadequate. Such learning outcomes or job 
training standard elements as “Be familiar with, . . .” 
“Understand the . . . .” “Operate the . . . .” and “Repair 
the . . . .” are not good enough. The analysis of learn- 
ing outcomes, which initially guides the development of 
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thw programmed instruction materials, must be inten- 
sive, extensive, and specific. Learning outcomes must 
define as completely as possible what performance 
the student is expected to demonstrate at the termina- 
tion of the program. The terminal behavior desired at 
the conclusion of the program must be stated in con- 
crete and explicit terms. There can be no misinterpre- 
tation by anyone as to the nature of the desired learning 
outcomes. It has been said that this condition is desir- 
able for all education and training programs. True. It 
is impossible, however, to develop an adequate pro- 
grammed instruction package without this. Program- 
med instruction forces this issue like no other instruc- 
tional technology we have developed. 

In summary then, what are the salient features of 
programmed instruction? 

1. The program begins with a specific description, in 
behavioral terms, of desired learning outcomes. 

2. The program is a carefully and logically arranged 
sequence of information. 

3. The student is required to be an active participant 
throu^out the program. 

4. The program is arranged so that every student 
can proceed at his own individual rate. 

5. The program must provide the student with im- 
mediate knowledge of the correctness of his responses. 

In the past, when students did not leam, and mate- 
rials of learning were not easily understood, we rarely 
if ever blamed the teacher or the textbook writer. The 
student more often than not was to blame. Either he 
was too stupid or too dense to understand our “simple” 
explanation or — and this has been a recent explanation 
in explaining away our failures as teachers — ^he was 
improperly educated by other teachers or schools. 

In programmed instruction an entirely new concept 
of responsibility enters the learning situation. The 
burden of responsibility for the student learning is on 
the program and the instructional technology used — 



not solely on the student. If the student doesn’t leam, 
something is wrong with the program. It must be 
thrown out and a new one tried. Or it must be “de- 
bugged” so that it will teach. Or it must be revised 
— ^the material must be re-sequenced or something else 
must be done to the program until it finally teaches — 
until it is such an excellent program that it teaches 
practically everybody. Some idealistic die-hiu:ds are 
willing to say everybody. But practically everybody 
means 95 to 98% of your target population. 

New vistas open up to the zealot of programmed in- 
struction. Students that other teachers and schools 
have given up on now become ripe material capable 
of cultivation. 

Programmed instruction is, in essence, a commit- 
ment — a commitment on the part of those who consider 
themselves teachers, instructors and trainers — ^to pro- 
duce materials that will insure learning in as many per- 
sons as is humanly possible. It is the kind of commit- 
ment made to learning that is made by any true 
professional to his mission accomplishment, regardless 
of what specific nature that mission may be. It is a 
final commitment to quality. 

Programmed instruction and automated teaching, 
rather than dehumanizing man, will humanize him. It 
is poor teaching, and poor, dull, and boring iustructors 
who have hidden behind routine and drill, that have led 
to much student failure and to the subsequent dehuman- 
ization of man. Programmed instruction offers learn- 
ing in the best socratic and tutorial tradition of teach- 
ing, asking that we bring difficult, stupid, unmotivated, 
and gifted though neglected students — “tempest-tost” 
by other teachers — and telling those students that the 
burden of learning is no longer solely on their shoul- 
ders, but that it is now equally, if not more so, on the 
shoulders of those who would consider themselves true 
professionals in the field of education and in what may 
someday become the science of pedagogy. 



II. Education 



THE EMERGENCE OF INSTRUCTIONAL TECHNOLOGY 

Gabriel D. Ofiesh, Colonel, USAF 

Colonel Gabriel D. Ofiesh, in the first presidential address to the society, points out 
that programmed learning is the first true system of instructional technology that 
assumes the major responsibility of learning, instead of placing it solely on the 
shoulders of the student. This educational technology, in common with other tech* 
nologies in other fields, not only hypothesizes learning experiences for students, but 
further provides for the constant testing and revision of the hypotheses to insure that 
students do in fact learn. It is the hypothesis, testing, and revision procedure which 
is so crucial in the development of programmed materials that makes possible the 
transition from the arts of pedagogy to the sciences of Instruction. It is in this that 
Colonel Ofiesh perceives the emergence of instructional technology. 



Programmed instruction has emerged as the first 
valid system of educational and instructional technology 
that our society has ever had. It is patently in error 
to categorize programmed instruction, as has so often 
been done, as simply another medium or technique or 
among the various audio-visual devices. It is more 
than this. Oron P. South (1962) Educational Advisor 
at the USAF Air University, has pointed out that: 

Continuous change introduces new dimensions into the 
problem of education and training. Whereas in the past we 
learned from stable systems, now we have to learn from 
rapidly and continuously changing systems. Old ways of 
learning and old patterns for learning may no longer be 
appropriate. 

When change comes slowly and almost imperceptibly, one 
can be clear as to both the process and content of education 
and training, as these are worked out over a long period of 
time. The relevance of what is taught, to whom, and how, 
is matched against reality. What is dysfunctional is dropped. 

As an innovation in educational technology, pro- 
grammed instruction, like no other approach that we 
have had in the field of education, is dropping essen- 
tially “what is dysfunctional” in the learning situation 
and developing what is truly functional. 

Many of the questions that are raised today regard- 
ing programmed instruction have been raised in the 
past in many other contexts. We have frequently 
heard the comment that programmed instruction is not 
new, that we have had it for years. It is probably 
valid to say that many of the characteristics of pro- 
grammed instruction, independently considered, are not, 
in and of themselves, new. Programmed instruction, 
however, has emphasized these characteristics with a 
lucidity and force that no other instructional technique 
has to-date. Programmed instruction has in essence 
“forced the issue” on many of these questions. Prior 
to programmed instruction many of these questions 
were largely academic. 



We should hesitate to say at this time that we have 
found, through programmed instruction, aU the answers 
to the ultimate development of an instructional tech- 
nology worthy of the name. Possibly through pro- 
grammed instruction as a vehicle we can see the first 
signs of a true emergence of instructional technology. 
It may be that we are so desperate for a technology 
that our anxiety is causing us to over-project the sig- 
nificance of these signs. 

Why is it that programmed instruction offers itself 
as such a vehicle? Why is it that programmed instruc- 
tion is raising the issue of instructional technology as 
nothing else has done to-date? If instruction is to 
truly gain substantive professional status in the years 
to come, then it will in all likelihood do so as it 
develops an instructional technology worthy of the 
name. 

If we ever develop a valid instructional technology, 
the teacher will gain prestige, because he will be able 
to actually structure productive learning experiences 
for students. Learning results will be so significantly 
superior that they will be convincing to all. 

Such an instructional technology will allow instruc- 
tors to be measured by the degree to which the student 
learns (not the degree to which the teacher “teaches”). 
There will be a realistic distinction made between 
those who “know the subject” (the scholars) and 
those who are able to take what the “scholars” know 
and put it into forms which are understandable to the 
student. We are not saying that those who do not 
know will teach. Nor are we saying that those who 
do know are necessarily effective teachers. The issue is 
too complex to be resolved by a simple statement or 
maxim. 

Technology has been described as an application of 
science to art. In essence, an instructional technology 
will seek to apply what we know of the science of 
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learning, along with what we may find out tomorrow 
about the science of instruction, to the art of teaching. 

When we hear criticism that the present system of 
teacher education is inadequate, or that teachers are 
being assigned to fields for which they are not thor- 
oughly prepared, we might ask what is really meant 
by “preparation”. Usually this criticism means that a 
teacher is ill-prepared in knowledge of the subject 
matter that is to be learned. Preparation involves not 
only knowing the subject matter, but also knowing how 
to structure for the student experiences out of which, 
it is hypothesized, learning will emerge and knowledges 
and skills will accrue. 

When a hypothesis is generated, the next step is to 
test it. The teacher is essentially a generator of hypo- 
theses. It is the failure to test the many teacher- 
generated pedogogical hypotheses that has kept instruc- 
tion a nebulous art rather than permitting it to develop 
as a science. We have been content to simply make 
educated guesses about the things we do as teachers. 
If the teacher were to test his hypothesis and find 
it wanting, then his only alternative would be to modify 
it and try it again, and again and again — ad infinitum. 
It is basically wrong, very wrong, to assume that the 
process of osmosis will transmit knowledge from the 
mind of the teacher to the mind of the student- 

There is a need for a great deal of experimentation 
in developing instructional technology. Such technology, 
if ever developed, must of course be subject to verifica- 
tion as to its adequacy. There is a common impression 
among many teachers that the most significant part 
of their teacher training was the practice teaching ex- 
perience, rather than the professional courses in educa- 
tion. This may or may not be a valid point. What is 
it about the practice teaching experience that gives 
this impression? To what extent does practice teaching 
provide the most effective training for the would-be 
teacher? Why do the usual courses in professional edu- 
cation give the impression, valid or not, that they do 
not have enough substance to train teachers in the 
skills of pedagogy? 

Would-be teachers are urged on the one hand to 
increase their knowledge in specific subject areas and 
at the same time to strengthen their backgrounds in 
general education so that they may be able to see the 
parallel developments in other subject matter fields 
which relate to their own particular specialized inter- 
ests. We cannot determine to any extent how any par- 
ticular process, or procedure, or course will improve 
teaching ability until we are able to define in more 
precise terms than we have to-date the specific skills 
of pedagogy and how they can be validated. We must 
be willing to measure teaching ability, not by what the 
teacher does, but by what happens to the student and 



how he ultimately performs. It is the student for whom 
we must structure the experiences out of which, it is 
hypothesized, he will learn. It is, therefore, the stu- 
dent’s performance that we must observe both prior 
to and following the learning experience. It is in 
observing this performance that we will determine how 
successful the teacher has been. 

There have been, among other things, for example, 
indictments of institutions of higher learning for using 
graduate students to teach elementary freshmen courses. 
This indictment may be well taken. One questions, 
however, whether it is deserving when very often a 
graduate student may have technical knowledge of 
pedagogy which his scholar-professor-advisor may not 
have. One should not question whether a graduate stu- 
dent should teach a freshman his elementary course in 
a particular subject matter. One should question 
whether either the graduate student or the scholar- 
professor deserves to teach the course because of his 
lack of ability to communicate to the student what it is 
that he wants to teach. One questions whether the 
graduate student who aspires to become a scholar, and 
would like to spend most of his time doing research 
and writing papers which are deserving of publication, 
would have less interest or inclination to empathize 
with the student than would the scholar. Both profes- 
sor and graduate student should be evaluated by their 
ability to teach students — in essence, to modify be- 
havior in a substantive manner. This, above all. 

We do not lack methods, ideas, procedures, or con- 
tent subject-matter knowledge. What we really lack is 
something substantive in the training of would-be teach- 
ers which will validate those skills which are success- 
ful in modifying in specified predetermined directions 
the behavior of students. We need both quantitative 
and qualitative measures of what happens to the student 
and how he performs. 

An instructional technology of any significance would 
be able to assess the difference between a successful 
learning experience and a relatively unsuccessful one. 
It would do even more than this. It would delineate 
several possible explanations as to why the one experi- 
ence was successful and the other one unsuccessful. 
The instructor must accept the commitment and ac- 
countability for the results of the learning experience. 
The instructor becomes the person who will organize 
the system of instruction and may himself not even 
be a part of it any more than the director of a play 
may feel the necessity to play any of the roles cus- 
tomarily delegated to the actors. 

An instructional technology worthy of its name will 
assist instructors in analyzing the structured learning 
experiences which fail to teach students. Deterline 
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(1962) provides us with some insight to the possible 
reasons why many instructors find diis difficult to do: 

In many cases even an experienced and skillful instructor 
will not be able to identify those subtle causes of a breakdown 
in instructional communication. An instructor might be evalu- 
ated by his students as being unable to communicate with them 
at their level, and it is likely that this criticism refers to the 
fact that most instructors, whether they are aware of it or 
not, make use of technical expressions, technical words, com- 
plex grammatical structures, and ambiguous or too-terse state- 
ments at a level of complexity that is not appropriate for 
the students. Obviously, an instructor does not have to 
operate at a too-technical level at all times in order to be 
guilty of this fault, since contact with the students can be 
lost in only a few sentences and never regained. It is often 
very dull to an instructor to elaborate on a point that seems 
very obvious to him. One difficulty is that the instructor is 
often unaware of the causa of the breakdown of communica- 
tion and the basis for confusion. What is worse, from the 
point of view of instructional effectiveness in the future, is 
that very, very few instructors make any systematic attempt 
to analyze what happens in their classes. When an instruc- 
tor loses contact with his students, continues to lecture, dis- 
covers that he must back up and cover some objective a 
second or third time, and then finally does succeed in clarifying 
the presentation, establishing contact once again, he is naturally 
not much inclined to stop at that point, make notes on the 
characteristi.'is of his presentation that failed and those that 
were successful, but is, rather, at least as anxious as the 
students to move on to fresh, new topics. Most instructors 
who are proficient in the subject matter — which is to say that 
they have the relevant objectives in their own repertoires — 
often are ineffectual in their instructional presentations be- 
cause they do not know the material from the point of view 
of effective sequencing for instructional purposes, and/or are 
unable to be consistent in presenting the material at a tech- 
nical level that is understandable, and in doing so in ^mall 
enough steps with adequate repetition, examples, clarifying 
analogies familiar to the student, and so on. 

In looking at this problem many tangential issues 
have been raised. The question is not class size, teacher- 
student ratio, or the number of M.A.’s and Ph.D.’s on 
the faculty. The question is not the number of publica- 
tions produced by the faculty. The question is not 
whether the scholars know very much about their 
subject matter. The question is not so much whether 
the student is willing and bright and motivated enough 
to learn. The question is not whether he has been 
properly selected. The real question is, out of the 
learning experiences which we build for the student, 
what really happens? How has his behavior changed? 
What is his nature before he came to us and what is it 
after he leaves us as teachers? What is the real value 
of the learning situation created by the teacher for the 
student? What does it produce in the way of knowl- 
edge, skills, and attitudes? What knowledge, skills, and 
attitudes does the student now possess that he did not 
have before he entered the teacher’s zone of experi- 
ence? Until we evaluate our teaching effectiveness by 
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the product instead of the process, by what happens to 
our students and how they perform rather than by sec- 
ondary criteria, we will never get to the root of the 
problem. Teachers who know how to teach and do not 
know what to teach are as common as teachers who 
know what to teach but who do not know how to teach 
it. I would like to entertain the notion that both of 
these examples are equally deficient. A scholar who 
cannot communicate his wisdom is just as inadequate 
as a pedagogue who has no wisdom to communicate. 
There are professors in our liberal arts colleges today 
who could do well to visit some public school, and find 
out what are the requireinents of good teaching, as 
there are teachers and professors of education who 
could do well to visit the halls of liberal arts colleges 
and find in their staid text books what there is that 
should be taught. But in either case, the professor in 
his liberal arts ivory tower and the professor of educa- 
tion in his conventional, lock-step classroom, should 
both be willing to stand a test of saying, “Measure my 
effectiveness as a professor, as a teacher, by what hap- 
pens to my students — not by what I say, not by what I 
do, not by how I behave, not by how popular I may be, 
not by my scholarly interests and achievements, but by 
my students. I submit that you measure my effective- 
ness by what happens to my students, and let my 
students speak for me and my skills by how they be- 
have and perform, and nothing else.” 

Joseph Tucker has pointed out that if we cannot 
give deserving teachers merit pay because we cannot 
objectively differentiate between a good teacher and a 
poor teacher, then one might well ask how we distin- 
gui^ between a teacher and a nonteacher? 

There is no real quarrel with conventional instruc- 
tional methodology. Note that we have said “methodo- 
logy”, not “technology.” Rather, until the recent 
developments which have evolved with programmed 
instruction, there has never been an instructional tech- 
nology, as such, to speak. Even today, one cannot say 
with any finality whatsoever that we have in pro- 
grammed instruction a true instructional technology. 
It is merely that, through programmed instruction, one 
can see the first signs of only the probable emergence 
of instructional technology. It would be less than 
candid were we to state the issue otherwise. 

If we consider programmed instruction within the 
parameters of all the foregoing remarks, then it is evi- 
dent that programmed instruction is, at best, a system 
of instructional technology which forces the issue on 
many of the questions which have been raised. This 
is not to say that programmed instruction wiU provide 
the answers to aU the questions that need to be an- 
swered; nor is it to say that programmed instruction 
is in and of itself the instructional technology that our 
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society needs. Ii is to say, however, that the adherents 
of programmed instruction have concerned themselves 
with testing the experiences which they hypothesize 
will lead to successful learning. Programmed iustruc- 
tion has raised questions regarding the adequacy of 
present measures of learning experiences. Through 
programmed instruction we are questioning the value 
of present achievement measures and the extent to 
which they help us determine the effectiveness of the 
learning experiences. Programmed instruction has 
deepened the concern with retention of learning and 
with the sequencing of instructional content to men- 
tion only a few issues. 

There has been an understandable concern with the 
procedures and techniques of programmed instruction 
itself and the devices of the accompanying teaching 
machines. If we concentrate, however, on the specific 
features of this innovation in its present very primi- 
tive and crude form and do not focus our perspective 
on the underlying concepts of programmed instruction, 
then we are not likely to find in programmed instruc- 
tion the solutions to our problems of developiug an 
adequate instructional and educational technology. 
What is more important to us, at present, is the under- 
lying concept of programmed instruction as an instruc- 
tional technology rather than the excessive delibera- 
tions_about linear versus intrinsic programming, overt 
versus coveft'responses, teaching pachines versus non- 
teaching machines, etc. Programmed iustruction is a 
concept, not a set of techniques. 

Through programmed iustruction and teaching ma- 
chine technology we have found a vehicle which per- 
mits us to study the educational process and to iso- 
late the significant variables which will lead to further 
study. Programmed instruction and the accompanying 
teaching machine technology will provide a heuristic 
model for the development of instructional technology. 
Stolurow (1962a) pomts to this possibility in mention- 
ing the advantage that teaching machines have over 
conventional instruction for experimental and demon- 
stration purposes: 

One of the advantages which the teaching machine has over 
conventional instruction is that it freezes the instructional 
process during a study in a way which permits it to be used 
with many individual students. Furthermore, it isolates the 
instructional material and its plan of organization from the 
personality and related factors which heretofore have always 
appeared in an inseparable combination with the teaching 
method. The combination now can be studied as separate 
components to determine the relative effectiveness of the 



method independent of these other factors. Later, research 
studies can combine programming and presentation methods 
with the other factors, such as teacher personality, to determine 
how they interact with one another. Thus we see the possibility 
of making genuine progress, and, in fact, of using a truly 
scientific approach to study instruction. Through such an 
approach one should be able to develop a theory of teaching 
which could guide the classroom teacher, as well as the writer, 
in developing educational materials along more effective lines. 

Through programmed instruction we are starting to 
think about the application of the engineering approach 
to the field of education. This is not to mechanize 
education, but rather to develop educational technology 
to the point where we can shoot with a bullet instead 
of a shotgun. Programmed instruction has forced us 
to concern ourselves with determining what it is we 
are about and establishing quality control for the process 
of education. 

Through programmed instruction we have been 
developing a core of adherents who have committed 
themselves to students learning. Try-out students for 
programmed materials are sought in the circles of 
rehabilitiation, delinquency, drop-outs, and in penal 
institutions. Adherents of programmed instruction have 
said to school administrators, “Give us your failures. 
Let us try-out our programs on the students who have 
failed to learn.” This has been a refreshing change on 
the part of a group of persons involved in pedagogy. 
This attitude, this type of commitment in education, 
will, more than anything else spur the development of 
instructional technology. 

In summary, we can say that through programmed 
instruction we have started to put under very intense 
and comprehensive scrutiny the complete procef:r of 
instruction and learning. We are asking questions about 
the relevance of our educational objectives and learn- 
ing outcomes, the relevance of various media, and the 
compatability of conventional educational literature, 
to mention only a few. 

It is difficult to anticipate the form that the tech- 
nology of programmed instruction will assume in the 
years to come. We are at present in the first primitive 
and crude stages. Incipient in this concept, however, is 
the potential of an instructional technology that is 
beginning to emerge. 

Someone has said that the greatest problem in com- 
munication is the illusion that it has been achieved. 
Programmed instruction has begun to dispell this illu- 
sion; instructional technology eventually will dispell 
it once and for-ail. 
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PROGRAMMED INSTRUCTION AND THE PHILOSOPHY OF EDUCATION 

John W. Blyth 

Programmed instruction increases our ability to control behavior. In order to make 
this ability compatible with an acceptable philosophy of education we must determine 
who is to control behavior and for what purpose. John Blyth points out that one must 
not lose sight of the fact that In addition to intrinsic behavior, a philosophy of educa- 
tion must of necessity include the intrinsic behavior necessary to learn how to respond 
to the stimuli found in the world around us. Programmed instruction may be used to 
obtain^ the aims of a puppet philosophy of education based either on intrinsic or 
extrinsic behavior depending on the aims of the programmer, which we hope are in 
the direction of maintaining a free society. 



OBJECTIVES 

The most ardent enthusiasts about programmed in- 
struction are usually quite eager to employ traditional 
methods of communication in the form of published 
papers and talks to groups. The papers and t^ks, how- 
. ever, do not always show any transfer effect from what 
should have been learned in developing a program. 

All program writers have found it necessary to form- 
ulate the objectives of a specific program with consid- 
erable precision. To get the program started, it is es- 
sential to begin with something already familiar to the 
intended learners. The learning process then proceeds 
step by step toward the objective. In going tlirough 
the program, the learner participates actively and is 
reinforced as he proceeds. 

In this p^i^t^r, I shall start by stating my objectives 
and shall then present some steps leading toward those 
objectives. But the steps will have to be fairly large, 
since much ground must be covered in a short tim e. 
Unfortunately, it will not be possible to elicit overt 
responses and offer positive reinforcement to all correct 
responses. 

One further problem may be anticipated. There will 
probably be some people who will continue to emit the 
wrong responses, even after negative reinforcement. 
If the only reason for this is the strength of prior con- 
ditioning, then further remedial programs are called 
for to extinguish the wrong response and strengthen 
the correct responses. However, some programmers 
will undoubtedly claim that I am reinforcing the wrong 
responses and will offer counter evidence. This kind 
of disagreement is welcome, because it illustrates the 
fact that the question, “What is the correct response?” 
is not always equivalent to the empirical questions, 
“What response do people in fact make?” and “What 
response is reinforced?” 

It is the fundamental objective of this paper to show 
that there are many important questions about educa- 
tion that are not equivalent to, and cannot be translated 
into and reduced to, empirical questions of fact, or 
questions about behavior. In more general terms, 
the philosophy of education is not identical with, and 



cannot be reduced to, the psychology of learning or 
the science of behavior. In scientific inquiry, it is es- 
sential to formulate our questions in a way that makes 
it possible to answer them by appeal to observable 
data. But it is equally important to recognize that 
there are many very important questions that are not 
answered by scientific inquiry. 

Confusion on these matters works both ways. On the 
one hand, there are unwarranted claims that we now 
have a science of behavior which will provide scientific 
answers to all questions about education. On the other 
hand, there are many educators who are reluctant or 
unwilling to take advantage of the results of scientific 
inquiry because they believe mistakenly that this com- 
mits them to an unacceptable philosophy of education. 

There are widespread, although poorly defined, fears 
that we are committed to a kind of puppet philosophy 
of education if we admit that there are ways to control 
human behavior. These fears are augmented by the 
fact that determinism is an integral part of communist 
doctrine and the fact that total control of human be- 
havior is a deliberate aim of totalitarian governments 
of all kinds. It is believed that we must reject the 
view that behavior can be controlled in order to main- 
tain a democratic and humanistic philosophy of edu- 
cation. 

I hope to cut through some of the confusions arising 
from a failure to distinguish scientific questions from 
philosophic questions. Programmed instruction in- 
creases our ability to control behavior. This opens up 
opportunities for more complete regimentation of men 
in totalitarian states and for more complete realization 
of human values in democratic states. With better 
knowledge about how to control behavior, the impor- 
tant question is no longer whether behavior can be con- 
trolled. The important question is, “Who will control 
behavior and for what purposes?” 

PHILOSOPHIC AND SCIENTIFIC QUESTIONS 

The first step in a program usually starts with some- 
thing familiar. One thing familiar to aU programmers, 
and one of the few things on which they all agree, is 
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the fact that the first step in writing programs is a 
clear formulation of the objective. It is frequently 
urged that the specification of objectives should be 
given as a description of the terminal behavior to be 
produced by the program. 

By stating objectives in terms of observable behavi- 
or, it may appear that we are still concerned only with 
questions to be settled by scientific observation. But 
this is impossible. If we follow the recommended pro- 
cedure of sper'.yijg objectives before writing a pro- 
gram, there is, at that time, no program and hence 
no terminal behavior to observe. 

No one has seriously proposed a kind of actuarial 
programming technique in which one first writes a pro- 
gram and then administers the program to different 
groups to develop a table of terminal behaviors pro- 
duced. Pushing the analogy one step further, we 
might use a sort of mortality table to identify a target 
population. That is, we identify the target population 
by finding what group of people produce the intended 
terminal behavior. 

Programmers do not work this way, either in theory 
or in practice. They first specify objectives for the in- 
tended group of students. The program is then de- 
veloped and finally tested on the intended population. 
Observations then become possible to determine 
whether the actual terminal behavior corresponds to 
the intended behavior. 

In short, there is no scientific answer to questions 
such as, “What is the intended terminal behavior of 
the program one is about to develop?” There is, of 
course, a school of thought which v/ould reply that this 
question is a meaningless question, because it has no 
scientific answer. From this point of view, an answer 
to the question would be equally meaningless, so we 
would be left with no guide in developing a program. 
We can only charge blindly ahead writing frames and 
then see what happens. 

Questions concerning the objectives of a program 
are apparently meaningful for those who answer 
such questions before writing a program. They should 
be able to take the next step with me. Questions re- 
garding the aims or objectives of an educational sys- 
tem are like those regarding a specific program. They 
are not questions of scientific fact. They are questions 
in the area of the philosophy of education. No matter 
how much scientific information we acquire about the 
control of human behavior, there are important non- 
scientific questions concerning who should exercise 
control and the purposes to be realized by that control. 

The distinction I am making works both ways. Em- 
pirical scientific investigation of the factors influencing 
behavior does not answer philosophic questions about 
the aims of education. Nor does philosophic inquiry 



about the aims of education settle any questions of 
fact regarding the most eflBcient method of attaining 
those aims. 

THE NATURE OF MAN 

The steps leading to the discrimination of a philos- 
ophic question from a scientific question started with 
the acknowledged fact that programmers specify the 
objectives of a program before writing it. To take the 
next step, let us look at another aspect of the program- 
mer’s activity. 

Does the program writer make a choice between pos- 
sible alternative objectives? The answer to this quest- 
tion is ordinarily “yes.” The answer is “no” only in 
cases where the program writer is acting under orders 
from an editor or an employer. But in those cases, 
there is always someone in a position of authority who 
has made the choice. 

This apparent dilemma involves a deep-seated and 
persistent confusion. The confusion is generated by a 
failure to make a few fairly simple discriminations. 
One important discrimination is suggested by the case 
of the program writer who is acting under orders in 
selecting an area for programming. Since he is not 
making a free choice, it might properly be said that 
his actions are controlled or determined for him. If 
the action of the man under orders is said to be deter- 
mined, it appears that we must say that the action of 
the man who makes a free choice is undetermined. 
Thus, accepting the view that human behavior is deter- 
mined seems to preclude the possibility of free choice. 

At this point, an apparent conflict emerges for a pro- 
grammer who has made a choice, but also believes that 
human behavior is determined by antecedent causes. 
If his actions are all determined, he cannot understand 
how he can make a free choice. 

The verbal behavior just described should receive 
negative reinforcement, since it constitutes the wrong 
response to questions about free choice. The following 
steps lead to the correct response. 

In this age of rocketry and automation, some of the 
clearest illustrations come from new developments. 
Some rockets are launched to follow a predetermined 
course. Their trajectories are determined by the force 
and direction of the rocket thrust. The action of the 
rocket is entirely under external control. 

Somewhat more complicated rockets have a built-in 
guidance system. The guidance system may be preset 
to follow a predetermined path, or it may be designed to 
locate a moving target and follow that target until 
overtaking it. In the former case, the behavior of the 
rocket is predetermined in its entirety. In the second 
case, the behavior of the rocket is, in part, under the 
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control of an internal guidance system. Its course is 
determined, not by some completely predetermined se- 
quence of signals, but by the responses of the guidance 
system to signals sent out by the rocket and bounced 
back from the target. There are also rockets with more 
complicated guidance systems. Their systems make 
them partially responsive to ground control signals and 
partially independent of such external control. The 
guidance system continuously processes signals from 
the ground and feedback from signals sent out by the 
system. The behavior of the rocket is thus determined 
by the output signals from its own guidance system. 

For each type of rocket just described, let us ask 
whether there is any suggestion of action or behavior 
not determined by antecedent events. The answer is 
clearly “no” for each rocket. Next, let us ask for each 
rocket, whether any of the rocket’s behavior is inde- 
pendent of external control. In this case, the answer 
varies from one rocket to the next. The rockets vary in 
the degree of control exercised by external events in 
relation to the degree of control exercised by internal 
events. Those rockets with an internal guidance sys- 
tem exhibit a certain amount of self-control. Thus in 
modem rocketry we find clear examples of a situation 
in which each action of a rocket is determined by an- 
tecedent events, but it is also correct to say that some 
of their actions are free from external control. 

We have now discriminated two distinct questions 
which have been confused in the traditional controversy 
over determinism and free will. We can now resolve 
the apparent dilemma of the program writer who 
thought he had no freedom in choosing objectives, be- 
cause he believed all his actions were determined. Let 
us separate the question us to whether his behavior is 
determined from the question as to what determines it. 

Is the behavior of a human being making choices 
determined by antecedent events? The answer is 
“yes”. 

Is the behavior of a human being making choices de- 
termined by external or internal antecedent events? 
The answer to this question will be “yes” in some 
cases and “no” in others. We should also recognize 
varying degrees of internal and external influences. A 
human being is in constant interaction with many 
forces in his environment. Some of his actions are di- 
rectly responsive to external stimuli. But most of his 
actions are determined to some degree by the process- 
ing of stimuli by his internal guidance system. To the 
extent that this is so, he is acting freely in making 
choices. 

A philosophy of education is concerned with the aims 
of education. The formulation of these aims is neces- 
sarily affected by our view of the nature of man. The 
scientific study of human behavior presupposes that 
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human actions are determined by antecedent events. 
This hypothesis not only does not conflict with the for- 
mulation of a philosophy of education, it is indeed pre- 
supposed in such a formulation. Teachers always have 
and always will assume that their actions have some 
influence on their students. The notion of a disem- 
bodied free will which somehow causes uncaused ac- 
tions has never been a useful concept. 

The view that man’s behavior is determined is an 
important premise to work with, but it does not settle 
any of the important questions in a philosophy of edu- 
cation. Among other things, a philosophy of education 
will specify the aims of an educational system for in- 
dividuals who are to learn certain ways of behavior, 
and are to learn certain subjects. The individual being 
educated makes many responses to stimuli, but the in- 
dividual is not a mere aggregate of separate stimulus 
— ^response connections. It is not responses that go to 
school, but the individuals who make responses. The 
philosophy of education requires a more satisfactory 
concept of an individual than we obtain simply by 
studying stimulus — ^response — ^reinforcement patterns. 
Of course, the science of behavior also requires some 
such concept. 

The study of astronomy is quite different from the 
study of the behavior of astronomers. The satellite 
Mariner was not sent around the world to examine the 
behavior of astronomers. It was sent up to examine the 
behavior of Venus. Our knowledge of astronomy is 
certainly created by the efforts of astronomers. But 
the subject of astronomy cannot be reduced to the as- 
tronomer’s behavior any more than a piece of beef- 
steak can be reduced to the act of chewing. In short, 
the philosophy of education must go beyond the science 
of behavior in formulating a concept of the subject to 
be taught to individuals. 

And, finally, the philosophy of education is con- 
cerned with the aims of education. As indicated above, 
questions regarding aims and objectives are not scien- 
tific questions. A consideration of aims takes us back 
to the individuals whose aims are involved and forward 
to groups of individuals in society. 

A PHILOSOPHY OF EDUCATION FOR 
A FREE SOCIETY 

We are now nearing our objective of clarifying the 
relation between programmed instruction and the 
philosophy of education. We have taken the following 
steps: 

1. We have distinguished questions concerning the 
objectives of a program and of an educational system 
from questions concerning the most effective way to 
attain those objectives. 
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2. Questions about the aims of education are philo- 
sophical. Questions concerning the most effective way 
to attain those aims are scientific. 

3. We have discriminated the question as to whether 
human behavior is determined from the question as to 
the location of controlling factors. 

4. We have seen that the view that man’s behavior 
is determined is not incompatible with the making of 
choices or, incidentally, with a conception of individual 
moral responsibility. 

5. And, finally, we have seen that the philosophy of 
education requires several concepts going beyond the 
science of behavior. 

The significance of programmed instruction and all 
other technological developments for a philosophy of 
education lies in the increased control over the edu- 
cational process which is now possible. However, in- 
creased control over human behavior is like increased 
control over atomic energy. The mere power to control 
is morally neutral. Both kinds of control can be exer- 
cised for quite different objectives. 

In authoritatian societies, the goal of an educational 
system is to bring the control of individual behavior 
more and more under the direction of some central 
authority. In such a society, the locus of controlling 
events is external to most individuals. This is a puppet 
philosophy of education. The authorities pull the 
strings. The individuals are like the rockets with a 
guidance system responsive only to external control 
signals. 



In a democratic society, the goal of the educational 
system should bring the control of an individual’s be- 
havior more and more under a developing internal 
guidance system. The goal, of course, is not completely 
independent from external signals. The person who 
becomes entirely unresponsive to external stimuli re- 
quires institutional treatment. The goal of a demo- 
cratic free society is to maximize the areas of indivi- 
dual choice. The successful operation of such a society 
requires citizens educated to make decisions for them- 
selves, both individually and cooperatively. They must 
have a guidance system which is not unresponsive to 
external stimuli. But their overt responses to external 
control signals will be mediated through their internal 
guidance system. 

In short, in a democratic society of free individuals 
we need more, not less, knowledge about the control 
of human behavior. In particular, we need much more 
knowledge about how to improve the internal control 
systems of individuals. This suggests the direction in 
which further scientific inquiry is required. This will 
be a more difficult stage of inquiry because at least 
some of the determining events are not so readily dis- 
criminated and observed as overt behavior. 

Programmed instruction may be used to attain the 
aims of a puppet philosophy of education, emphasizing 
external control, or to attain the aims of individualistic 
philosophies of education, emphasizing internal control. 
Let us hope that we can program the applications of 
programmed instruction to maintain a free society. 



PROGRAMMED INSTRUaiON AND THE CONTROL OF BEHAVIOR 

William A. Deterline 

William Deterline points out that versatile free minds as a goal of our education 
system can only be achieved through a highly disciplined and controlled learning 
environment. He insists that the lack of control is more likely to produce undisciplined 
slipshod thinkers than creative unstifled individuals. He believes the American educa- 
tional system needs the greater discipline that programmed instruction can provide to 
produce men educated to their fullest capacity. 



I do not intend this paper to be a polemic aimed at 
education. The government commissions officers who 
are admirably qualified for that purpose. I do not even 
feel that my remarks are unjust criticisms of all edu- 
cational practices, even though I am about to make 
some disparaging remarks about many of the products 
of American education. First I intend to alienate al- 
most all educators by criticizing the fruits of their ef- 
forts. TTien I intend to alienate almost all advocates 
of programmed instruction, because I believe that pro- 
grams, as we write them now, do not do the things 
we ascribe to them. Our programs are little related 
to our laboratory techniques of shaping behavior, or to 
our favorite old saw, the tutor-student interaction. In 
spite of this approach I hope that the tone of this paper 
will not be a completely negative one. I do believe in 
the potentialities of programmed instruction, and I wiU 
take exception to the cautious and conservative state- 
ment that programmed instruction is not a panacea for 
the problems of mass education and training. I firmly 
believe that in our present crude art lie the seeds of 
just such a panacea. 

“Control of Behavior” is a phrase that seems almost 
to be in direct antithesis of our idealized educational 
objectives of developing “free minds” and of provid- 
ing a liberal or liberating education. There is little 
doubt in my mind, however, that we achieve that edu- 
cational objective only when our instructional method- 
ology exercises an extremely tight degree of control 
over the behavior of the learning organizm. This ap- 
parent contradiction in terms, production of a versatile, 
freely adaptive thinker through extremely tight and 
rigorous control of his thinking during learning, is ac- 
tually not a contradiction at all, but a blurring of the 
distinction between means and ends. One is the end 
and the other is the means. Because we have never 
in the past completely exercised these means, we have 
very rarely achieved the desired end. 

Generally speaking, the product of our formal edu- 
cational institutions, through college, is not a liberally 
educated student but a sparsely educated one. Instead 
of turning out “free minds”, we produce unliberated, 
undisciplined slipshod thinkers whose primary educa- 
tional distinction is the impressive way that they have 
slipped through 12 to 16 years of instruction without 



thoroughly mastering any subject. Certainly we want 
creativity and adaptability, and many classroom teach- 
ing techniques give the student freedom to attempt 
creativity and free expression. But aU too often these 
activities are unguided, uncontrolled, and, as a result, 
actually interfere with the desired learning. Our high 
school and college graduates tend to have an impress- 
ive capability for impractical, unrealistic, ineffective 
and uninformed guess work that is, in part, the prod- 
uct of a lack of disciplined basic intellectual skills. 

We admire the educated guess and deplore the blind 
stab, yet we do not consistently produce educated stu- 
dents who are capable of educated guesses. Why not? 
Too often during a student’s academic career, he never 
learns the things to which he is exposed; his response 
repertoire is not permanently changed in the desired 
direction. 

We are proud of our educational system, and, up to 
a point, we should be. We do have effective mass pro- 
duction of sparsely educated people despite the short- 
age of good, dedicated teachers. Something is wrong, 
however, and the examples that should bother us are 
quite plentiful. During the late 1930’s and early 1940’s 
tile United States built the atomic bomb, and we used 
physicists and chemists and engineers from many 
other countries to do it. Look at the great names that 
were involved in the Manhatten Project; names of men 
at the very top of their fields from all over the world. 
Only a few of the very great were products of Amer- 
ican education. I read recently that Leonard Bernstein 
is the only native-born conductor of a major symphony 
orchestra in the United States. I don’t know whether 
this is true, but even a much larger proportion would 
still be appaling. There are many more examples like 
these that indicate that, per capita, American educa- 
tion is not turning out enough top flight scholars. These 
examples are not cited in order to compare U.S. edu- 
cation with education in other countries, but only to 
point out the apparent waste of raw material in this 
country. 

What do we accomplish in our schools? We do de- 
velop a great deal of social conformity, but we do not, 
unfortunately, develop enough academic conformity 
in subject matter areas. All of our high school graduates 
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may be able to tell us that the sum of 2 and 2 is 
4, but beyond this level, conformity in demonstrable 
knowledge and performance is extremely limited. We 
should strive for conformity in certain basic methods, 
facts, relationships, and necessary broad objectives of 
instruction. We should produce a more uniform prod- 
uct at every grade level, and we cannot blame all the 
variance on innate individual differences- Industry 
and the military services spend many millions of dol- 
lars every year in what amounts to remedial training 
of high school and college graduates. These students 
missed too many fundamentals that should have been 
acquired by all students. Granted, our classrooms are 
crowded, good teachers are in short supply, and the 
curricula are often crowded with non-essentials, but 
our greatest deficiency, I feel, is the extreme lack of 
uniformity of proficiencies of our students at all lev- 
els. And this leads directly to my original topic, the 
control of behavior. 

As an organism responds, he learns. Learning is in- 
efficient when the learner is given no steady support, 
no continuous guidance, and no continuous stimulus in- 
puts that require specific response outputs. We lec- 
ture to students but fail to try to control their thinking 
about the subject matter so that only accurate thought 
patterns occur. In effect, we turn the students loose 
and hope that they will think the way we do. We want 
the thinking of the students to emulate the thinking of 
the teacher, but rarely do we show them how we think; 
and rarely is the desired behavior shaped in the stu- 
dents with any kind of uniformity. Instruction is still 
too much an osmotic process m which the learner is 
the sorter. He must organize and select stimuli, and 
produce responses without adequate guidance. The 
student’s success as sorter and structurer of knowledge 
is never very certain, because we force him to play 
the game of trial and error. When conventional in- 
struction is criticized because of its lock-step pacing, 
its failure to insure continuous active student respond- 
ing, its complete lack of immediate reinforcement, and 
so on, I think we are only identifying facets of the much 
larger deficiency, lack of control of learning. 

Programmed instruction provides a measure of con- 
trol of beharior beyond that currently a part of any 
conventional instructional methodology. Programs 
may never reduce the number of teachers in our 
schools (although programmed texts may replace most 
of our unprogrammed textbooks), but programs will 
certainly provide a degree of controlled behavior that 
will replace much of the uncontrolled behavior that 
currently fills the student’s time. By this it is meant 
that a program will unerringly guide a student from the 
beginning of the study of a topic directly to the objec- 
tive of that topic without losing control of the student’s 



behavior. We say that students must be actively in- 
volved as they use a program, but this is not that big 
an innovation. Some students are passive some of the 
time, of course, but that passivity is far less detrimen- 
tal than the inappropriate active responses that carry 
the student into unrelated topics or into incorrect con- 
clusions and extrapolations. 

Uncontrolled active responding is far too common 
in many of our classrooms. Active responding by it- 
self is not necessarily desirable, unless optimum con- 
trol is exerted over the stimulus environment and over 
the direction and content of the behavior sequences. 
We can make use of programs to control student be- 
havior by keeping learning guided in a direction that 
reliably produces the desired finished product. The 
bad work “uniformity” may rear its ugly head, but 
we do want to produce certain uniformities of knowl- 
edge. The creative individuality of a college student 
who spells Pennsylvania “P-e-n-s-i-l-vania” or who 
demonstrates complete lack of comprehensible con- 
formity in expressing himself in 'writing, or in solving 
a simple problem in mathematics, is not a desirable 
activity. It is also not very individually creative for a 
college student to define “indivisib^ as “something we 
cannot see.” I have seen each or these examples of 
non-uniformity, but I don’t think that this is the kind of 
creativity we seek. 

A student can be creative and original only if he is 
prepared for it. Instead, we allow creativity as a teach- 
ing method to interfere with and even preclude the pos- 
sibility that the student will ever achieve a solid back- 
ground preparation for creative judgement. 

It was Newton, not Mac the cobbler, who built us a 
physics. The contributors to science, technology, the 
arts and humanities have not been the partially trained: 
they have been individuals thoroughly grounded in 
both basic principles and the more abstract forms in 
their knowledge areas. Integration of new knowledge 
is relatively easy for one who is thoroughly prepared, 
but extremely difficult for the dabbler. We now prin- 
cipally turn out dabblers who know very little. Our 
colleges are probably even more guilty of this then are 
the public schools. 

College students encounter some very rudimentary 
statistical concepts in general or educational psychol- 
ogy, and panic generally ensues. These students were 
only dabblers in their mathematics courses and their 
behavior was probably so extremely uncontrolled 
through elementary and secondary school that they 
never became reliably competent in mathematics. Vast 
numbers of college students wail plaintively, “I’m hav- 
ing trouble with this statistics chapter because I have a 
mental block about mathematics”, and we must admit 
that both competing emotional responses and competing 
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intellectual responses do occur. These blocks stand 
in the way of successful learning of the statistical 
concepts, formulae and operations. Unfortunately, 
students who make the claim of mathematics mental 
blocks do so almost smugly because our society con- 
dones a satisfaction with academic incompetence in the 
difficult areas of math and science. 

Students who are not competent at lower level skills 
or fundamental stages of understanding are severly 
handicapped when higher levels are assigned. Students 
who proceed through a good program are competent 
at each step in the program and this competency wiU 
contribute to the elimination and prevention of mental 
blocks. Trial and error and learning to avoid previous 
mistakes, Norman Crowder norwithstanding, are not 
the most effective and efficient learning processes. 

My emphasis on greater control of the learning proc- 
ess is subject to the same criticisms that have been lev- 
eled at programmed instruction for other reasons. L. J. 
Cronbach, one of our leading educational psychologists, 
has criticized programmed instruction severely, feeling 
that our current instructional methods are good enough, 
having stood the test of time. Other educators have 
echoed these sentiments. But time is a poor test. Aris- 
totle’s assertion that heavy objects fall faster than 
lighter objects stood the test of time for 2000 years 
until Galileo tested it. Old theories of matter satisfac- 
torily stood the test until 1900 when they became inad- 
equate and had to be drastically revised in order to 
allow science to enter the nuclear age. Uncontrolled 
instructional technologies must make room for a tech- 
nology that will add a greater degree of control to the 
learning process in order to allow the system to become 
more effective. Teachers have always felt twinges of 
futility when correcting examination papers, because 
so many students demonstrate so many deficiencies by 
their inability to answer questions. Greater control 
through programmed instruction may not be the whole 
answer, but programmed instruction can certainly 
provide a tremendous contribution by controlling learn- 
ing, thus helping to produce graduates with a more 
thorough and less variable repertoire of proficiencies. 

There is only one thing wrong with this picture, and 
that is the problem of validity of our existing analysis 
of how and what programmed instruction should do. 
We say that programmed instruction came from the 
psychological animal laboratory. We also draw the 
analogy of programmed instruction with the Socratic 
or tutorial interaction. These extrapolations are ele- 
gant and enlightening, but they may be false, and this 
is an alarming possiblity. 

In teaching motor skills it is easy to control the stu- 
dent’s behavior because we can see what he is doing 
or is about to do. We can guide his behavior, evaluate 



it continuously, and stop any missteps before they have 
more than begun. Intellectual learning is not visible; 
we can never really know what kind of covert respon- 
ses are occurring unless they have an overt counter- 
part. One result of this difference is that we can pro- 
duce a skier or a swimmer or a sculptor faster than 
we can teach intellectual skills. We can guide and 
control motor activity, but thinking is not visible enough 
to tell us how to guide or what controls to apply. Think- 
ing continues even while a student reads a frame and 
responds to it. So far we have not been able to get this 
behavior completely under control. Sometimes this 
thinking helps the student move forward, but just as 
often it may lead to errors, misconceptions, negative 
transfer, and so on. Our animal analogies from the 
laboratory are based on motor skills and may have very 
little to do with human conceptual learning! 

I intentionally overstate the degree of my concern 
with the possibility that the behavior we call thinking 
may not be adequately controlled or controllable by a 
program. I assume that thinking behavior follows the 
same principles and obeys the same laws as other as- 
pects of behavior. The success of our programs has 
given support to this view. If this assumption is cor- 
rect, it means that we can eliminate the most undesir- 
able aspect of human learning in our schools: the un- 
guided, uncontrolled trial and error activity that the 
student emits in response to the presentation of mate- 
rial that is to be learned. 

The student faced with the problem of adding to his 
repertoire those responses that will adequately handle 
problem situations in the classroom, on tests and in 
application outside the classroom, needs aU the guid- 
ance and control that we can provide. When his be- 
havior is not guided and controlled, much of his 
activity goes unreinforced or is followed by aversive 
consequences. As we would expect, the result is ex- 
tinction or avoidance learning. Students do not learn 
the subject matter, and what is worse, develop a 
“mental block” for that subject or for academic learn- 
ing in general. Mastery and competence, as we use 
these terms, provide the basis for continual reinforce- 
ment of successful performance. My main point is 
this: losing control of the student’s behavior as he 
learns means that we cannot prevent undesirable learn- 
ing; incorrect answers and emotional responses such 
as frustration and avoidance will occur. When we con- 
trol his learning we insure, not only the acquisition of 
the desired responses, but also the prevention of un- 
desirable behavior. 

It should be clear that I am not advocating class- 
room control in the sense of stifling self-expression, 
reducing the adaptability and flexiblity of the teacher, 
or reducing the classroom interaction to a regimented. 
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non-social and “no fun” experience. Only a small part 
of the student’s time is spent in the acquisition of knowl- 
edge and skill, but this activity must be controlled as 
carefully as possible if instructional objectives are to 
be achieved. The advice given to beginning golfers and 
skiers, to employ a skilled instructor so that no bad 
habits can be learned, is the equivalent of advising the 
learner to obtain a competent source of control of the 
learning process. 

There is an aura of smugness apparent in many ar- 
ticles and papers that reflects a general feeling that 
programmed instruction has arrived! There are cer- 
tainly impressive experimental and applied results, but 
programmed instruction has not yet arrived at any com- 
fortable and assured place in our instructional frame- 
work. I feel, as do many psychologists, that these very 
crude beginnings in the application of behavior theory 
to the practical needs of society need to be recognized 
as being crude. 

In the past the classification of programs as either 
linear or branching reflected two dilBferent attitudes 
toward control of student behavior. Early branching 
programs, I feel, exerted too little control over the 
student; his thinking was too lightly controlled be- 
tween the questions that separated the units of textual 
material. After leamiug had or had not occurred, the 
program reacted appropriately, letting the student pro- 
ceed or correcting the error that the program had di- 
agnosed. Early linear programs, on the other hand, in- 
sisted that all students must fit a single mold program, 
encountering exactly the same stimulus sequence and 
following exactly the same behavioral sequence. Nei- 
ther of these approaches has been completely adequate. 
A branching program was able to adapt with some 
flexibility, but exercised too little control, while a lin- 
ear program provided almost too much control since 
it could not in any way adapt to the developing capabil- 
ities of the students. 

The use of various kinds of branching, bypass, wash- 
back and other types of adaptive sequencing has made 
basically linear programs into much more powerful 
instructional materials. Unfortunately, we too often 
lost sight of the very behavior that we were trying to 
shape, became enamored of a technique and, like our 
programs, remained rigid and inflexible. 

Programs must include adaptive sequencing in order 
to maintain an adequately tight control over the stu- 
duet. If the interaction between a student and a pro- 
gram cannot bend, it will break. I think the worst 
offenders in failing to provide adaptive control have 
been the Skinnerian experimental psychologists, of 



whom I am one, who have transferred their attention 
from the systematic production of behavior in animal 
laboratories to the production of human behavior via 
programmed instruction. In the animal laboratory, 
electronic control gadgetry, called programmers, are 
used to provide all stimulus inputs to our experimental 
equipment and the subjects inside, and to provide all 
appropriate stimulus consequences of the subject’s 
behavior. Even the electronic equipment has a great 
deal of adaptability built into it, manipulanda that 
react to a wide range of response intensities, topogra- 
phy, direction and so on. Latching relays and delay 
circuits are also used to provide for certain acceptable 
latitudes in response latencies and frequencies. Even 
so, some of our subjects will not adequately develop 
the response patterns wanted, so we often resort to dis- 
connecting the programming devices and using the ex- 
tremely adaptable method of successive approxima- 
tions in order not to lose a subject. Unfortunately, in 
preparing programmed mstructional materials we 
have not been equally adaptable. 

Control must be maintained but should not be con- 
fused with an inflexibility of control that can often be 
as disruptive as too little control. We have already 
strayed too far away from the student and although we 
give lip service to his role in developing programs, we 
too often do not use him imtil after the basic sequence 
is written and all formats are established. We do not 
now allow the student to control the programmer as we 
should. What I mean is that we psychologists, acade- 
micians and training specialists have already turned 
our backs on a basic premise of programming — that 
the program must be written to interact effectively 
with students and that this can best be accomplished 
by having students participate actively in the program- 
ming process right from the beginning of work on the 
program. 

Programming principles, as noted earlier, are based 
on laboratory procedures for producing motor behavi- 
or sequences in lower animals. There is an appallingly 
small amoimt of ejq)erimental data dealing with the 
analysis of these principles applied to hun?an concep- 
tual learning. Only a little research is being done that 
will eventually provide us with additional techniques 
which can adequately control the human learning proc- 
esses. Only when tight, predictable, reliable control of 
learning becomes an available technological tool will 
psychology have made its contribution to education. 
Only then will education have some hope of becoming 
a scientific technology rather than an art that is prac- 
ticed well by only a very, very few. 
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PROGRAMMING THE IMPOSSIBLE 

David J. Kiaus 

David Klaus points out that in the early stages of any developing technology, 
attempted predictions often prove to be significant underestimates: He draws ana- 
logies and concludes that much which is now considered impossible in education will 
be achieved through programmed instruction. 



One of the most disturbing aspects of programmed 
instruction is the amount of sensationalism surrounding 
it. The terrors of mechanical teachers offering mass- 
produced, push-button education is the starting point 
of most articles to be found in the daily press, in Sun- 
day supplements, in popular magazines, or even in trade 
publications. Unfortunately, every headline suggesting 
robots in the classroom is followed a short time later 
by the statement of a responsible local educator that 
no dramatic change is contemplated and that parents 
may feel assured that their children will not be sub- 
jected to any such radical revision in instructional prac- 
tices. Rarely does the public hear of the legitimate 
successes of competently conducted investigations. In- 
stead, there is a tendency to publicize the guarded com- 
ments of educational conservatives who continue to 
regard the potential of programmed instruction with 
considerable skepticism. 

Historically, even the most optimistic predictions con- 
cerning any developing technolo^ often prove to be 
significant underestimates. What is impossible one day 
may well be commonplace the next. This is certainly no 
less likely to be characteristic of programmed instruc- 
tion than of any other technology. For the researcher, 
the impossible becomes an objective, a challenge. 

Because such goals as producing students who are 
able to judge fine art with sophistication, uniformly able 
to create original proofs to geometric theorems, solve 
problems in calculus before they are ten, or learn a 
language so thoroughly that all of them master all the 
vocabulary and grammer given, are not within the cur- 
rent capabilities of education and training institutions, 
does not mean these goals can never be achieved. The 
researcher must try not only to better understand how 
learning occurs and how it may be enhanced, but to 
use this knowledge i.i attaining desirable objectives as 
well. In a sense thtn, programming the impossible 
could never occur unleVs there was a universal willing- 
ness to accept the present status of education as one 
which defied improvement. Because a legitimate goal 
has not been attained before does not mean it is for- 
ever beyond reach. Only ^’O long as there are defi.- 
ciencies in education will there be room for new tech- 
niques and only so long as there are those willing to 
say some one goal is inherently .unattainable will it be 
po.ssible to program the impossible. 



In another sense, programming the impossible can 
never be a reality; there always wiU be new problems 
to overcome. Yet, it is worthwhile to look at some 
past and present views of programming with the hope 
of stimulating program users, and possibly program 
producers, to seek new horizons and not dwell com- 
placently among their existing, and unfortunately rein- 
forcing, accomplishments. It must be remembered that 
in discussing programmed instruction, it is all too easy 
to over-generalize and suggest that more has been ac- 
complished than the findings warrant. Those vast 
improvements which were often predicted for program- 
ming have not fully materialized, at least as yet, and 
are not likely to do so without considerably more re- 
search. Nevertheless, programming already has made 
significant progress, at least enough to cause concern 
to the traditionalists and offer hope to the revolutionists, 
and much more can be expected. 

An important consideration of programming research 
is that the problems to overcome are typically the prob- 
lems of education in general. Parents, teachers, and 
administrators, to say nothing of the vast majority of 
students, agree that every effort should be made to 
develop instructional resources that are capable of 
producing more efficient, effective, and economical 
learning. The dedicated educator is far from pleased 
unless all of his pupils learn as much as possible as 
rapidly as possible. He may well be skeptical of magi- 
cal formulae and unwilling to adopt new techniques 
without thoroughly investigating their advantages and 
limitations. Such an educator does not perceive pro- 
gramming as threatening, nor is he anxious to defend 
the mediocrity he himself is capable of identifying in 
many current educational practices. 

The first reaction to programming, now rarely heard, 
is that it simply cannot work. Students are too vari- 
able, too inattentive, and too much dependent upon 
skilled supervision to ever profit from printed mate- 
rials, regardless of how they are presented. Programs, 
at least early programs, did not appear too terribly 
different from textbooks or workbooks used regularly 
in the schools. But students could and did learn from 
programs, a finding perhaps more significant than 
any other thus far obtained. The systematic control of 
learning by means of carefully prepared and sequenced 
learning experiences was shown to be not impossible; 
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it was shown that the materials and the outcomes they 
produced easily and conveniently could be replicated 
wherever and whenever desired. This is not to imply 
that programs, especialy the early ones, were not with- 
out shortcomings. These programs often had limited 
applicability and frequently had errors in content. They 
were frequently tedious to all students and fully suc- 
cessful for only a narrow range of them. But they 
worked. To many programmers, this remains the most 
important development of all. Unfortunately it is a 
result which is frequently overlooked by those anxious 
to focus attention on whatever programming has not yet 
been able to achieve. 

That programmed instruction could successfully as- 
sume the burden of instruction where other media often 
fail was an impossibility only to those who have not 
kept pace with the rapid growth in knowledge of the 
learning process. Until recently, most texts dealing 
with educational psychology carefully distinguished 
between human learning and conditioning studies. Far 
greater emphasis was devoted to findings on serial- 
position effects and the meaningfulness of material than 
to reinforcement and stimulus discrimination. The 
function of the classroom teacher and textbook writer 
alike was to inform, demonstrate, explain, and some- 
times discipline; but it was the student who had to 
study, learn, and remember. Successes were credited 
to the capable and energetic teacher; failures to the 
dull and inattentive student. Programmed instruction 
was capable of producing a degree of control over 
learning that was beyond the scope of any other form 
of instruction concerned primarily with the presentation 
of information. Regardless of whether they taught bet- 
ter or faster than more traditional forms of instruction, 
the fact that programs did teach and could assume the 
burden of instruction remains an impressive and still 
important finding. 

The second impossible objective sought by many 
programmers was the development of a technology 
which was capable, on empirical grounds, of surpassing 
achievement standards that are generally regarded as 
satisfactory in the schools. The suggestion that properly 
prepared materials could be expected to lead to ninety 
per cent of the students learning ninety per cent of the 
content was made during an orientation course for mili- 
tary training personnel in response to a class member’s 
question. While it was tempting to reply that a goal of 
everyone learning everything seemed reasonable, it also 
seemed reasonable to make some allowances for inade- 
quacies in the state-of-the-arts. At that time, only a 
small fraction of prepared programs were this satisfac- 
tory, and more than one member of the class felt free 
to demonstrate his belief that this goal was impossible. 
Yet, by subsequently insisting upon that level of quality 



in its own extensive programming effort, the Air Force 
has shown this to be a realistic goal; one which un- 
doubtedly could be attained by programs developed for 
other educational use as well. 

Uniformly high standards of achievements are almost 
an inherent property of programmed materials. Given 
enough patience, the program writer could subject his 
sequences to as many cycles of tryout and revision as 
are required to produce any desired attainment. Cer- 
tainly, the systematic use of empirical data to improve 
a unit of instruction is not exclusive with programming. 
Yet, it is diflScult to explain why such an objective as 
ninety per cent of the students learning ninety per cent 
of the content was thought of as impossible. Perhaps 
it was the recklessness of enthusiastic zealots, as some 
programmers are described, or perhaps it was the con- 
fidence of educators and training specialists that the 
standards established could not easily be achieved 
which led to this goal. Whatever the reason, however, 
the uniformity of high standards of achievement which 
can be produced by programs undoubtedly will have a 
lasting impact on many facets of education and em- 
ployment. 

A third impossibility which is being overcome con- 
cerns programmed materials which are capable of deal- 
ing with the wide heterogeneous range of individual 
talents found in a typical classroom or with the pecu- 
liarities of special groups. For years, the individual has 
been an outstanding concern in education; yet, except 
for the limited adjustments made possible by multiple 
tract systems and special classes, it was not possible 
to provide the individual attention so strongly recom- 
mended. Similarly, it was thought to be impossible for 
programs which have uniform content and which are 
administered with no provision for student questions to 
provide for the peculiarities which make students differ- 
ent. In making this assumption the ease with which 
programs adjust to variations in learning ability and 
student background by means of individual pacing 
through branching, by-passing, and similar techniques 
was largely ignored. The seemingly overwhelming 
problem associated with students completing a course 
of instruction at different times was not seen in proper 
perspective; no one asked why it is important for aU 
students to finish together even if they are not equiva- 
lent in achievement. It should be apparent that anyone 
clever enough to cope with graduates having achieved 
a random mastery of the subject in constant time can 
successfully deal with graduates all having similar 
achievement but learned in varying amounts of time. 

One paradox of programmed instruction which re- 
flects this impossibility is the pride with which many 
teachers point to the use of programs in dealing with 
students needing remedial instruction. How strange it 
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is to hear of the success of a program when applied to 
the most difficult cases from educators who are reluc- 
tant even to try the same program for the regular in- 
struction of the remaining students. A second paradox 
concerns the concept that students will be molded to 
an unnecessary degree by programs. If this concept is 
used to raise an objection to giving the dullest in a 
group the opportunity to be alike in knowledge learned 
to the very brightest students, then much will be gained 
by having greater uniformity in instruction. On the 
other hand, if the objection is meant to imply that it is 
impossible for each student to learn something different, 
it should be evident that programmed instruction will 
make it feasible to offer all students a choice of lan- 
guages, sciences, and other subjects to an extent not 
possible in most schools today. Required courses were 
inserted in the curriculums, hopefully not to facilitate 
checking attendance, but rather to produce students 
who were uniformly successful in learning uniform 
content. 

Other individualized aspects of learning, such as 
motivation, are certainly problems. While it has not 
been impossible to make programs interesting as well 
as instructionally effective, this is an area in which 
much more progress can be expected. One significant 
aspect of the approach being used by most program- 
mers is that learning need not be difficult to be good. 
Why learning should be considered an effort, and de- 
scribed as schoolwork or homework, has never been 
made clear. Certainly, entertainment is not the objec- 
tive of education; yet, instruction must compete with 
other opportunities offered to the student and should 
be at least not unrewarding or unpleasant. It should 
not be impossible to devise programmed materials which 
will incorporate new techniques or material necessary 
to attract learners. Perhaps, someday, research will 
reverse the present situation and the circus performer, 
television producer, and fishing supply salesman will 
look enviously at the educator, and apologize for his 
own poor showing by expounding knowledgeably on 
the problems of motivation. 

The fourth impossibility it that the rules of animal 
learning might be reasonably valid when applied to 
human performance, the principles of conditioning, 
regardless of which set might be employed, could be 
used to enhance human learning. It should be clear, 
however, that even if premises such as insight, reflec- 
tion, rationality and other supposedly human qualities 
are granted (and many behavioral scientists are unwill- 
ing to grant such premises) it has been possible with 
programming to produce more efficient and effective 
learning than has previously been accomplished through 
mass media. Programmed instruction represents one 
of the first opportunities to systematically produce or 



engineer desired changes in human behavior. Perhaps 
this forebodes an uncomfortable societal problem since 
the control of skills and factual knowledge is not far 
different from the management of concepts, attitudes, 
and opinions; and since these are not much different 
from the manipulation of advertising or propaganda to 
produce lasting and profound changes in politics, com- 
merce, ethics or fundamental beliefs. 

The rules used in preparing programs are presently 
inconsistent, inaccurate, and far from inclusive. Yet 
enough is known to produce reasonably rapid kinds of 
change in generalized responses or what is generally in- 
terpreted as changes in attitude or judgment. Although 
attitudes, and similar forms of complex behavior, are 
not taught deliberately, they are certainly learned. It 
should not be too far in the future when programs will, 
in small deliberately planned increments, provide the 
opportunity to acquire a particular attitude toward some 
political party, economic institution, esoteric organiza- 
tion, or consumer product. Commentators, public re- 
lations specialists, and advertisers represent an expendi- 
ture of considerable time, effort and money. It is going 
to be a quite interesting event when the manufacturers 
of Brand Y, that seemingly unlimited line of inferior 
products, fight back with a program that is ninety per 
cent effective for ninety per cent of the consuming 
public. 

One of the most interesting impossibilities has been 
the work on content. The origin of programniing seems 
characterized by a combination of scientific modesty 
and not much optimism. While it was agreed that pro- 
grammed instruction would be beneficial wherever ex- 
tensive practice with rote materials was required, there 
was little except theory to suggest that elaborate con- 
cepts, aesthetic judgment, creativity or similar skills 
would ever be reduced to programming. Today, how- 
ever, work on many of these areas is in progress. Not 
only does it seem possible that these complex kinds of 
performance can be taught, but it is likely that the pro- 
cedures used in preparing programs will generate truly 
capable graduates with far more assurance than present 
methods. In other words, it now does not seem impos- 
sible that those objectives central to modem concepts 
of education could be taught effectively and efficiently 
by programmed means. 

Why anything but simple factual knowledge, and 
perhaps well-defined principles seemed beyond the ca- 
pabilities of programming unfortunately reflects a belief 
that the processes of education and training must re- 
main forever outside the realm of empiricism. Instmc- 
tion has been the object of far too little research and 
far too much discourse. Mediocrity has not been elimi- 
nated from the classroom despite the extent to which it 
has been challenged by those teachers and administrators 
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who have been seriously critical of the accomplishments 
of our instructional systems. For instance, the classroom 
teacher hopes to produce creativity and similar course 
outcomes; but she is typically at a loss to describe what 
she does to assist the student in being creative, aside 
from being permissive and not imposing too much struc- 
ture. Yet, she feels this and similar categories of in- 
struction are her areas of greatest competence and those 
which could never be dealt with by programmed instruc- 
tion. Instilling routine facts, on the other hand, is a 
task at which most teachers are successful; so much so 
that activities of this kind are not considered a chal- 
lenge. But, just why this level of instruction, which can 
be presented successfully by a teacher, should be allo- 
cated to programming is not at all clear. 

Programmers, just as teachers, are rarely excited by 
the opportunity to provide practice exercises, in arith- 
metic, language vocabulary, or irregular spellings. Al- 
though there is considerable discussion as to the out- 
comes desired from practice on these subjects, the 
methods for assuring mastery are both varied and well 
known. The chances of producing a truly significant 
advance in instructional technology for rote subjects 
(one which will produce a substantial increase in the 
effectiveness of instruction) are relatively remote. On 
the other hand, the possibility of utilizing the methods 
inherent in programming to discover some reliable pro- 
cess for instilling insight, developing interpretive skills 
or producing appreciation represents a very realistic 
goal for the programmer. 

If the methods employed by the programmer were 
typically used in the classroom, it would not be so 
unusual to consider creativity or originality in the same 
light as physics or history. To begin with, emphasis 
must be placed upon the product and not the process 
of instruction. Active discourse between the teacher 
and pupil is frequently identified as one reason why pro- 
grams will never entirely replace teachers. Yet, the 
objective of classroom discussion is never made too 
clear; it is supposed to be inherently a good thing and a 
desirable educational process. Active discourse is also 
supposed to promote critical thinking and analytic skills. 
Because this objective is certainly a reasonable one for 
education and for many kinds of training, the program- 
ming strategy which might be used to achieve critical 
thinking and analytic skills will serve as a suitable ex- 
ample of the approach taken in programming complex 
behaviors. 

It is important to determine the characteristics of 
criterion performance first; that is, how it can be ob- 
served and the ways in which it might be assessed. 
There are many who might contend that critical think- 
ing and analysis cannot be objectively defined, described 
or measured. The failure to establish what is meant by 
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these skills is an admission of defeat since without some 
means of determining the presence or absence of rele- 
vant responses (regardless of how subtle they might be) 
there is no way of establishing that the success of in- 
struction is greater than no instruction at all. This 
predicament applies equally to the classroom teacher 
and the programmer; neither is in a position to verify 
the effectiveness of instruction without some means of 
observing the outcome. Criteria performance can be 
described objectively in a variety of ways. One might 
look at the conclusions drawn by the student after being 
exposed to an argument, the sequence of questions or 
comments he verbalizes during a discussion, his analysis 
of written material, or any other student behavior, pro- 
duct, or combination of behaviors and products. Which 
objective or group of objectives is selected is only im- 
portant to the nature of the outcome, not the method 
of instruction. 

Once the objectives have been decided and once 
some empirical procedure for determining whether or 
not they have been achieved has been selected, the 
programming process can begin. To devise an instruc- 
tional sequence, the programmer attempts to utilize 
what the students already know and gradually, by 
adding only one increment of skill at a time, promote 
increasingly acceptable performance. In the case of 
critical thinking and analytic skills, the programmer 
might begin with any of a number of strategies, trying 
one after the other, until he is convinced (on the basis 
of the empirical evidence of the success or failure of the 
materials) that he has not only an acceptable but fairly 
optimal approach. He does not decide on a process 
first and then stick with this process regardless of the 
results obtained from student tryouts. Instead, he is 
willing to concede the inadequacy of his first, second, 
third, or even fourth attempt so long as any significant 
portion of his tryout students fail to exhibit the desired 
terminal performance. 

One strategy the programmer might try is the class- 
room technique of active discussion. There are at least 
two reasons, however, why he might not give much 
priority to this approach. First, he has no reason to 
believe the technique has led to more than token or 
chance success in the classroom. Second, the approach 
is unwieldy, not easily automated, and difficult to im- 
plement. Instead, the programmer is apt to concen- 
trate on an approach which can be individualized to the 
extent that each student can independently proceed 
through the instruction at his own pace and without 
outside assistance. These requirements are not neces- 
sary; of course, they may make his task a more difficult 
one. However, the programmer is likely to aim toward 
this form of instruction because of its potential econ- 
omy and wide applicability. 
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It is not at all necesaiy to duplicate the process of 
the classroom to produce the same product. In pro- 
gramming critical thinking, one approach might be to 
begin, not with the production of an argument, but with 
instruction designed to enable the student to discriminate 
between good and poor arguments. After this judg- 
mental skill has been perfected, erroneous or inadequate 
statements of increasing difficulty can be given to the 
student for him to edit. Finally, he is given the oppor- 
tunity to prepare an argument from the beginning. This 
procedure of developing skills in the sequence of first 
judge, then editor, and last producer is the reverse of 
those generally used in instruction. In technical writing, 
for example, only the best writers become editors and 
only the best editors become judges. By reversing the 
usual ordering of skill development, it is possible to 
instruct a writer who initially prepares a passage and 
then continues to edit it until he is satisfied with the 
passage in terms of the judgments he has already 
learned. This approach, which is only one of several 
the programmer might try, has the additional advantage 
of permitting each student to maintain an individual 
style of writing, since he has learned how to judge 
productions of many different styles and reflecting a 
vaiiety of content. Every student could be different, 
but well within the range of excellence established' as 
criterion performance. 

An instructional process similar to the one described 
could be duplicated by the classroom teacher; but the 
difficulty inherent in instnicting many students at once 
would largely preclude individual rate of progress or 
the systematic improvement of instruction on the basis 
of the details of student tryouts. At best, the teacher 
would identify and carefully record what responses ihe 
student should make at each point in the instructional 
process, what kinds of assistance seemed best, what 



practice material should be used to insure the greatest 
possible breadth for the skills being learned, and what 
sequence appeared to be optimal in presenting these 
various components. The resulting collection would be 
a draft program which, considering the amount of work 
that had been devoted to it, should then be tried out 
and made available as the program it is. 

Programming is far from a magical process; it cannot 
solve problems other than those it is capable of solving. 
Instruction, v/hether offered in the classroom, by an 
individual tutor, over television, with books, or through 
programs, is basically an effort to arrange the student’s 
environment so that the desired learning will occur as 
easily and rapidly as possible. Learning can occur 
under almost any set of circumstances; in fact, large 
amounts of learning take place without the assistance 
of formal instruction of any sort. Experience, which 
might be defined as the making of relevant responses 
under appropriate circumstances, is learning. The con- 
tribution of instruction is to arrange for the necessary 
kind and amount of experience without leaving it up to 
chance. Programming simply is an effort to condense 
the necessary experiences as concisely and systemati- 
cally as possible so as to insure learning will occur. 
Viewed from this framework, no learned human be- 
haviour is unamenable to programmed instruction and 
the benefits it potentially affords. 

In conclusion, programming the impossible has only 
been made possible by the fact that the effectiveness 
and efficiency of instruction are not all most people 
wr. It them to be. Programming is far more than a tool 
of education and training. It is a new approach which 
has the promise of providing all students the oppor- 
tunity to gain more from the instruction they receive so 
that they are better prepared to cope with the increas- 
ingly complex world in which they live. 
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A DIALOGUE BETWEEN TEACHING AND TESTING 

Thomas F. Gilbert 

Thomas Gilbert stresses the paradoxes that achievement tests can be both valuable 
and useless at one and the same time, and that traditional teaching methods can both 
increase and decrease individual differences. He clearly illustrates that a “test of 
knowledge” should be sensitive to many different aspects of learning, not just one 
cut-and-dried textbook ritual which is stamped “knowledge”. He shows that acquisi- 
tion of knowledge is often overlooked if application of certain techniques in demon- 
strating this knowledge is faulty. Too many students are graded on their expert or 
inexpert application of these techniques; their vast knowledge of a subiect cannot be 
applied, and the result is “failure” of a course of study. 



Tests of achievement pn'sent a paradox. On the one 
hand, traditional tests appear to obscure individual 
differences; on the other, traditional tests seem to mag- 
nify and exaggerate individual differences. The paradox 
has a solution, however, and it shows both cases to be 
true. The solution also demands that we depart from 
traditional procedures for constructing and interpreting 
our tests. 

Tom, Dick and Harry are three real and not imagi- 
nary students. We give them a test of long division, a 
subject in which all have had instruction. The test has 
very high reliability and validity, and the stamp of 
approval of 50 licensed psychologists. It is adminis- 
tered under the most careful conditions. All three boys 
are at the peak of attention. Tom and Dick both score 
50 percent on the test, and Harry makes a perfect zero. 
Obviously, Tom and Dick represent individuals who 
do not differ, and Harry clearly doesn’t know a thing 
about long division. 

Odd, for I happen to know that Harry can perform 
with speed and accuracy more than 99 per cent of the 
operations of long division, and I also know that Tom 
and Dick missed entirely different problems on the test. 
And no one, neither Tom, Dick, Harry, nor our 50 
learned doctors, cheated. 

What happened is simple. Tom failed half of all the 
long division problems. Dick failed the other half, and 
Harry knows every operation of long division but one: 
when he sees a problem stated, 75 — 15, he persistently 
makes the error of putting the dividend outside the long 
division bracket and the divisor inside. In every other 
respect his performance is perfect. 

Now we give the test to Harry’s twin brother, and he 
scores 100 per cent. This completes the demonstration. 
Our test has obscured the great differences between 
Tom and Dick, and it has exaggerated the small differ- 
ence between Harry and his brother. There are no tricks 
here. How can Tom do any division if he doesn’t know 
all the operations? When he multiplies and get a prod- 
uct that is a multiple of ten, he is confused because he 
hasn’t learned to treat zeroes like other numbers: “put 
down zero and carry one” doesn’t make sense to him. 



although he is at home with “put down 3 and carry 6.” 
Nor can Tom subtract from zeroes. Dick, on the other 
hand, has a different set of problems. He doesn’t 
know what to do if the first subtraction gives a re- 
mainder which the divisor goes into less than one whole 
time. You can see that there are all kinds of ways to 
fail at long division, but all of them do not apply to 
every problem. Dick knows some of these ways, and 
Tom the others. Do not listen to the arguments that 
Tom, Dick and Harry are merely unusual statistical 
events and that the test works on the whole. Nonsense! 
That has nothing to do with our problem and is only 
the old siren call to the bliss of statistical ignorance. 
There is a purpose for which our test is nearly perfect 
and of great potential use. There is another purpose 
for which it is useless and, indeed, can be damaging 
and downright subversive to the ends of education. 

So we have a test that both obscures and exaggerates 
individual differences. Is the test a good one? The 
answer is both a decisive “yes” and a definite “no”. 
We will soon see why, for there is a more general rule 
that w'ill permit us to resolve paradoxes such as these, 
and that will help us make more intelligent decisions 
about the use of educational instruments. 

But first we shall examine two more apparently con- 
tradictory propositions: (1) traditional teaching meth- 
ods increase individual differences: (2) traditional 
teaching methods decrease individual differences. 

The first of these propositions should be easy to de- 
dend; I could even appeal to the authority of the 50 
learned psychologists. I believe that 20 years ago it 
was the right answer on a psychology test to say that 
practice in learning a skill increased individual differ- 
ences, and recently I discovered that that answer is still 
graded as correct. With training the smart evidently 
get smarter and the dumb get dumber, at least by com- 
parison. Let Tom, Dick and Harry illustrate. 

After their long division test it became clear that 
Tom and Dick had potential, but 50 per cent was in- 
sufficient realization of it. Fortunately for Tom, the 
teachers discovered his difficulty and began to give all 
the low scoring students drills in handling zeroes in 
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arithmetic. Tom and Dick worked many long division 
problems requiring subtraction of zeroes. Of course 
Tom had to struggle while Dick breezed through the 
extra drills, but Tom finally understood zero subtrac- 
tion. Poor Harry was socially passed, and his twin 
brother began to experiment with new ways to do long 
division, for he added when he should have subtracted, 
and multiplied when he should have divided, and he 
tried various combinations of number games. Some of 
these superstitious games gave him by chance, for the 
first time, right answers, and naturally he took to these 
games as a race track tout takes to curious betting 
combinations. The result: Harry ceased to know long 
division. 

After a time our test was administered again. Tom 
made a 100 per cent, having mastered his zero prob- 
lems; Dick made 50 per cent again, and you can ima- 
gine his surprise and depression since he had whizzed 
through the practice problems which Tom had struggled 
over. Harry improved, getting 10 per cent, and his 
scratch paper looked like a numerologist’s notebook. 
Harry’s brother was excused from the exam since he 
had been placed in a special class for oveixated students. 
The standard deviation of the test has increased since 
the first administration, yet this statistic fails to reflect 
the even greater differences produced by our teaching 
scheme. Surely this successfully defends the proposi- 
tion that teaching can exaggerate individual differences. 

We see still greater individual differences by examin- 
ing what eventually became of the three boys and 
Harry’s brother as a result of their division training. 
Dick’s 50 per cent so affected him that he quit school 
and became a juvenile delinquent, cheating the society 
that cheated him. Harry’s superbrother ran into a snag 
in calculus and gave it up because his superiority in 
long division was comfortable enough; he ended up as 
a government bureaucrat who gleefully scrutinizes re- 
search proposals for failures to meet this or that me- 
chanical formality, particularly with respect to divisions 
of the budget. One hundred per cent Tom became a 
small business man who now appreciates the slow 
accretion of numerical gain. Poor Harry pursued his 
hapless attempts to solve long division and would have 
become an astrologist had his family not been opposed 
to the occult. In time, Harry became a mathematical 
genius, for his strange play with numbers led him to 
develop whole new formulations of mathematics that 
require, in their parsimony, no long division at all. 
Alas, he still cannot perform long division, and his 
twin brother has never let one of his unusual research 
proposals get by that critical bureaucratic eye. Cer- 
tainly, individual differences have been produced. 

The account of the teaching of Tom, Dick and Harry 
isn’t imaginary, nor is it atypical. However, this ac- 
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count of the resulting careers of Tom, Dick and Harry, 
although not imaginary, is atypical. A more represen- 
tative account will defend the second proposition that 
traditional education discourages individual differences. 

Actually it came to pass that Dick achieved 100 per 
cent in the next year. Harry finally got 100 per cent 
in his fre.shman year at college. And Tom’s earlier 100 
per cent victory over Dick was a Pyrrhic one; as he 
struggled with long division he missed out on dependent 
clauses and semi-colons, which he didn’t master until 
his senior year. Harry’s twin brother, however, did 
run into a snag in calculus and was able to find con- 
solation in his achievements in long division and 
sentence diagramming. 

For all three boys, and Harry’s brother, there op- 
erated a common factor. The rales of the mechanics of 
reading, writing, and arithmetic became the focal points 
of success and failure in the world of adult rituals. The 
boys all passed from expressive childhood into oppres- 
sive adulthood, from scrawling on bathroom walls to 
reading printed regulations that grownups hang in pub- 
lic places, and they came to believe that a man who can 
not speU well can not write well. Native wisdom 
came and went and inchoate knowledge lingered. In 
short, they all became bureaucrats, government or 
otherwise. That is what really happened. Individual 
differences were discouraged. So hard it is to teach the 
human the elementary that he tends to become dedi- 
cated to it, and obliterated is the individuality potential 
in his otherwise superior animal goods. 

Paradoxes exist only when we are talking ab^ at two 
different things as if we were talking about the same 
thing. This is precisely the cause of the paradoxes we 
consider here. There are two ways in which individual 
human repertories can differ, and tests can be sensitive 
to one and not to the other; teaching can affect one 
without affecting the other: (1) we can measure the 
number of operations missing from, or existing in, a 
behavior repertory, or (2) we can measure the effects 
a behavior repertory has upon the environment. I will 
illustrate. 

Harry made zero on his first test, yet he knew 99 
per cent of the operations of long division. If the test 
were measuring what he knew about long division, if 
it were assessing the operants established in his reper- 
tory, he would make a score of 99. Clearly the test 
was not measuring the operations of mastery achieved 
by Harry, although it is called an achievement test. 
We have already examined the grave consequences that 
fell upon Hariy when his teacher interpreted his test 
score as a valid measure of what he knew about long 
division. 

But look at Harry’s test score from another point of 
view. The test was graded by counting the number of 
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correctly completed problems. For all of Harry’s skill 
in long division, he was unable to produce the final 
product required of mastery; his achievement was as 
useless as that of one who had achieved nothing at all. 
He had acquired much but accomplished nothing. Re- 
garding the social usefulness of Harry’s repertory, the 
effect it can have upon the environment; the zero test 
score is perfectly valid. Anyone who needs right an- 
swers from long division would have no business em- 
ploying Harry, and if the test score had reflected what 
Harry knew about long division rather than the value 
of his knowledge, it could have badly misled a potential 
employer. 

It should be clear from this that the makers of 
achievement tests owe it to their customers to tell them 
what aspect of the behavior repertory the test scores 
distinguish. I suggest that achievement tests carry one 
of two names to indicate this distinction. Acquirement 
tests would yield scores that reflect the degree to which 
the operations of mastery have been acquired; accom- 
plishment tests would measure the value of a reper- 
tory. (These two words are the synonyms of achieve- 
ment appropriate for the required distinction. Acquire- 
ment connotes achievement produced by learning; 
accomplishment has the added connotation of acquire- 
ments that are useful in social intercourse.) Our long 
division test is clearly one of accomplishment, but few 
achievement tests are this pure, for they uncritically 
confound the two aspects of individual differences. 

Let us examine other examples of the confusion 
between acquirement and accomplishment. You say 
that you know nothing about bank tellering; but this 
is not true. If you examine all the operations a bank 
teller uses to balance his books you will find that you 
have already acquired most of them. Nevertheless, the 
few operations you have not acquired prevent you from 
accomplishing the objectives of the teller’s job. In 
terms of what the teller and you know about his job, 
there is very little difference between you; in terms of 
the value of what you know, you are worlds apart. I 
say I know nothing of marksmanship, but examine my 
behavior when I attempt to imitate a sharpshooter: so 
small are the differences between my actions and those 
of the master that I can fool you until I fire the gun, 
and even then I don’t miss very far. But a miss is as 
good as a mile, and the value of my repertory is miles 
apart from that of the sharpshooter’s. 

A small change in acquirement may produce an 
enormous change in accomplishment. A minute change 
in the way I grasp a golf club may more than double 
the effectiveness of my repertory. And if someone had 
shown Harry that he was confusing the dividend and 
divisor, he would have immediately become a division 
expert. The kid who failed physics knows almost all 



there is to know about physics, but his accomplishment 
is worthless. 

Now let us examine the meaning of individual differ- 
ences for educational practice. It should be an agree- 
able conclusion that our long division test is not 
appropriate as a guide to teaching, and that it is poten- 
tially quite dangerous. Education is charged with 
producing accomplished citizens, but acquirement is the 
means to this end. Here the egg must precede the 
chicken, for acquirement is the only route to accomp- 
lishment. It may be a cause for concern to some that 
measures of accomplishment predominate in educa- 
tional achievement tests. 

The most important error fostered in educational 
thought by the failure to distinguish systematically be- 
tween acquirement and accomplishment has been the 
overrating of individual differences in acquirement. 
Among students of similar age and culture, differences 
in acquired repertories may be small, as they usually 
are when compared with differences in accomplishment. 
But accomplishment impresses us more than acquire- 
ment, and the effect of confusing the two is to make us 
believe that differences among behavior repertories is 
much greater than it actually is. Since a technology of 
education must concern itself directly with acquire- 
ment and not accomplishment, it follows that we are 
likely to suppose that a genuine technology is extremely 
difficult to attain, for it is easier to arrive at a tech- 
nology if its subject matter is relatively homogeneous. 
It is because behavior repertories of human beings are 
strikingly more similar to each other than we are ac- 
customed to think that a technology of teaching was 
easier to develop and came about earlier than we had 
expected. 

Reconsider individual differences among long division 
repertories. While Tom and Dick missed half the prob- 
lems on the test, they actually are able to perform many 
more than half the operations. The operations of long 
division are numerous, if we consider those behaviors 
which must be separately learned. They consist of such 
things as writing the number six, writing the number 
seven, treating all two digit numbers as larger than all 
one digit numbers, treating the number three as larger 
than the number two, treating the number nine as 
larger than the number five, substracting zero from 
four, subtracting eight from ten, multiplying three by 
14, estimating the partial quotient of 89 when divided 
by 17, setting up an algorithm, distinguishing the tens 
column from units column, arranging successive esti- 
mates of the quotient from left to right, and so on. The 
student who fails to acquire any one of these operations 
will suffer serious consequences for his ability to dem- 
onstrate his accomplishment. But we can see that Tom 
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and Dick, who made 50 per cent on their test, must 
have acquired 95 per cent or more of the separate 
operations. So Harry, who knew 99 per cent of divi- 
sion, made a test score of zero. If we had known pre- 
cisely what it is that these boys did not know, it would 
have been much easier to teach them than we had 
expected it to be. 

What I have said is that differences among behavior 
repertories are small and that, at any given age and 
school grade, the students will already have acquired 
most of the elements of mastery of a course of study. 
Added to this, the components of mastery which the stu- 
dents have not acquired tend to be much the same from 
student to student (a little thought will convince the 
skeptic of this). Now we have a picture of individual 
differences quite at odds with the traditional concep- 
tion. Our subject-matter — the repertory of students — 
is considerably more homogeneous than we are ac- 
customed to think. The differences are tiny, but the 
effect of these tiny differences can be dramatic; and we 
are more likely to see the effects of these differences 
than the causes of them. 

A third consideration will lead us to view individual 
differences in acquirement as almost, though not quite, 
inconsequential. If we go back in time to where Tom, 
Dick and Harry first began to study long division we 
will see that the differences among them were even 
smaller than they were after instruction. They all 
would have made zero on the division test at that time. 



All knew how to estimate quotients, all knew how to 
multiply and substract, except that Tom couldn’t handle 
some zero problems. Dick and Harry are ready for 
instruction and Tom needs only a small amount of 
work on subtracting zeroes. \^en Tom’s remedial 
work is accomplished, we are prepared to instruct 
them. Analysis tells us that the boys all know nearly 
everything about long division except the order and 
format in which they are to carry out the acts of esti- 
mating quotients, multiplying, and subtracting. It is 
important for this point to be very clear: if the boys 
are ready for long division instruction, they each know 
all the component acts of long division, and it is rea- 
sonable to assume that none of them know the correct 
sequence of these acts. With respect to this subject 
matter the boys are virtually identical; Tom is unable 
to subtract as fast as the rest, Dick can estimate quo- 
tients faster, Harry is more rapid in multiplication, and 
Harry’s brother is less careless than the others and per- 
haps more eager; but all know the component opera- 
tions and none know the division sequence. Their 
differences are a small part of the total repertory being 
considered. As nave seen, these small differences 
can have great consequences if we aren’t careful. And 
care in education need not be as difficult as it seems. 

What happens if we are not careful? Just as we have 
seen. We will teach the boys in such a manner that 
they will differ greatly in accomplishment but only 
slightly in acquirement. 



PROGRAMMED INSTRUCTION COMPARED WITH AUTOMATED INSTRUCTION 

Norman A. Crowder 

Norman Crowder, in his usual provocafive style, points out that about the objective 
of programmed instruction there is unanimity: more effective and efficient learning; 
about the means, confusion, and often invidious contention, reigns. The basic differ- 
ence rests with the purpose of student responses. The linear approach views student 
responses as essential to the learning process and is thus greatly concerned with the 
sentence-by-sentence structure, or the microstructure, of the education process. 
Intrinsic programming, on the other hand, deals with the macrostructure of instruc- 
tional materials; student responses are feedback mechanisms which determine the 
course of the educational process. As such, intrinsic programming is a truly automated 
process since presentation of material is directly controlled by the activity of the 
student. Linear programming is indeed automatic but not automated since what the 
student does, does not affect the course of the process. 



Every now and then a well-meaning acquaintance of 
mine will say to me, “Norman, why is it that you per- 
versely insist on quarreling with the linear program- 
mers? Why don’t you seek a compromise, a middle 
position that will accommodate all point of view? Or 
better yet, why don’t you just agree with everybody?” 

For the benefit of these peacemakers, who presum- 
ably otherwise would not get their reward until they 
get to heaven, I want to report that I recently had the 
pleasure of listening to Dr. Skinner and agreed heartily 
with something that he said. What he said was that, at 
the present time, the field of programmed instruction is 
in a state of appalling confusion. I think this is true 
and I am anxious that this confusion be ended. It is 
just for this reason, however, that I have not placed 
great emphasis on finding middle grounds or median 
positions that will represent aU points of view. To do 
so would be quite misleading. The way to end the con- 
fusion is to state each of the various viewpoints clearly, 
pointing out and contrasting the differences in approach, 
assumption, and resulting technique. Surely within the 
scientific community one could expect to make such 
comparisons without being accused of invidious intent, 
or even to argue for the greater appropriateness of one 
assumption over another without being considered 
merely contentious. At least one hopes this is so, for it 
is only oy such comparison and contrast that the con- 
fusion, of which Dr. Skinner is not alone in complain- 
ing, can be ended. 

There are a great many different things being talked 
about under the heading of programmed instruction. 
All of the people engaged in the field seem to have as 
their objective a better educated student, usually with- 
out much intercession by a live instructor in the educa- 
tional process. There are wide differences, however, in 
the things that people seem to feel important in bring- 
ing this result about. Thus, for example, I find Susan 
Markle quoted as saying that the distinguishing feature 
of programmed instruction is the try-out of materials. 
Now certainly it is characteristic of programmed in- 



structional materials that they are easier to try out than 
are other prepared instructional materials, but this 
characteristic does not define for the uninformed what 
the processes are that are relied upon to induce 
learning. 

Similarly, I have the impression from Robert Mager 
and others that they are more concerned with clearly 
specifying the objectives of the instruction than with the 
techniques of instruction. No one will quarrel, of 
course, with the desirability of clearly specifying educa- 
tional objectives, and programmed instruction does 
make it easier to correlate the objectives with the actual 
instruction. But the desirability of clearly stated educa- 
tional objectives is not peculiar to the field of pro- 
grammed instruction. 

Related to those who place great emphasis on the 
clear setting of objectives are those who see pro- 
grammed instruction as a means of introducing curricu- 
lum reform, and rely on the improved curriculum to 
produce superior results. Again, it is true as a practical 
point that programmed instruction is a convenient 
means of introducing curricular changes, since the re- 
quirement for retraining of instructors is lessened. It 
is also true that the detailed attention that the subject 
matter receives in being programmed may lead to cur- 
ricular reorganization; in fact, it usually does. But the 
genius of programmed instruction does not lie in cur- 
ricular revision. 

Finally, we find in the literature a number of what 
I would call propaganda definitions, such as: “Pro- 
grammed learning is the organization of instructional 
materials in sequences optimally organized to produce 
student learning”. Such a definition is impressive, but 
it does not define very well what principles are to be 
used in the organization of material, or what process 
is to be relied upon to produce learning. I wonder, 
when I read such definitions, what teacher ever delib- 
erately organized his materials in any other way than 
the one which he believed would optimally promote 
student learning. 
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All of the things I have just discussed — the opportu- 
nity for careful preliminary try-out of materials, the 
possibility of defining educational objectives clearly and 
of making the instruction consonant, point by point, 
with these objectives, the opportunity to revise the cur- 
riculum independent of a supply of experienced instruc- 
tors, the occasion and opportunity for detailed analysis 
and reorganization of the subject matter — are valuable 
ancillary features of, or auxiliary benefits to be derived 
from, programmed instruction. But the defining fea- 
ture of programmed instruction does not lie in any of 
the things so far discussed, but in the process whereby 
the instruction itself is to be brought about. But it is 
here, at the heart of the subject, that we find the great- 
est difference of opinion among current practitioners of 
the art. Hence, the confusion in the field. It was with 
the intention of making the basic theoretical difference 
quite explicit that I chose as a title for the presentation 
“Programmed instruction compared with automated 
instruction”. 

All forms of programmed instruction depend on the 
individual student’s being more successful in using the 
newly prepared materials than he has been with con- 
ventional text materials. To bring about this more suc- 
cessful independent use of materials, two quite different 
possibilities have been set forth to be exploited, one 
chiefly by Professor Skinner and the other primarily 
by myself. 

Professor Skinner proposed to exploit the present 
work in learning theory as a guide to the improvement 
of what we might call the microstructure of the instruc- 
tional materials. Specifically, he proposed to so struc- 
ture the instructional materials that, at each step in the 
process, explicit and systematic use is made of existing 
response tendencies to elicit the “correct” response to 
new stimuli. This correct response, so elicited, is con- 
firmed and thereby reinforced and so learned and be- 
comes part of the now-existing response repertoire 
which may be used in the next step, and so on. 

Presumably I need not elucidate this model further, 
since it is the basis of the linear style of progr ammin g 
and so is familiar to all; I only wanted to point out that 
the characteristic feature of linear programming is an 
attempt to improve the sentence by sentence effective- 
ness of written communication, the microstructure, if 
you will, of the learning process. Beyond the peculiar- 
ities induced by the requirement to elicit and reinforce 
the student response at each step in the process, the 
resulting linear materials are, in effect, conventional in 
structure and intended use. 

I have dealt elsewhere with my reasons for believ- 
ing that improvement in instructional materials, which 
is limited to the micrqstructure of those materials, can 
meet with only limited success, and will not rehearse 



those reasons in detail here. Basically it comes down to 
the problem posed by human variability. The more 
carefully we fit the microstructure of instructional mate- 
rials to a single individual, the less good the fit must 
be for a second individual. We can attain the illusion 
of fitting everyone if we judge only by the student’s 
success in giving the correct responses, since in general 
a good student can do what a dull student can, but 
surely this is naive. But I stray from my point, which is 
this: what I had proposed to exploit to improve the 
effectiveness of instruction was the possibility of testing 
the student quite frequently and using the test result to 
determine what to do with him next. The intrinsic pro- 
gramming technique, which in essence is the automatic 
use of the student’s response to direct him to appro- 
priate material, makes it possible to manipulate the 
macrostructure of the educational process completely 
and automatically with a device as simple as the 
“scrambled book”. 

Numerous advantages arise from this ability to auto- 
matically vary the instruction on the basis of the stu- 
dent’s response. Direct evidence of the student’s readi- 
ness to attack a new point is obtained at each step, 
independent of the student’s evaluation of his readiness. 
Anticipatible errors can be provided for in the answer 
choices provided and appropriate remedial material pre- 
pared and automatically presented. Most important of 
all, in my opinion, is the fact that the questions set for 
the student may be of any level of difiiculty we judge 
to be desirable, i.e., since we have a means to deal spe- 
cifically with errors, we need not fear that errors will 
be made and thus can write programs which exert an 
upward pressure on the student’s abilities, rather than 
forcing all students through the path determined by the 
dullest. But again, I am straying from my main theme, 
which was the sources of confusion in the present dis- 
cussions of programmed instruction. 

The basic difference, which so often goes unrecog- 
nized, between linear and intrinsic programming is the 
fact that each is concerned with a different part of tlie 
educational process. The linear programs are directed 
to improving the microstructure cf the process, i.e., 
they seek methods whereby the student can be made 
infallibly to learn at each step. The intrinsic program- 
mers are not convinced that the microstructure can be 
sufficiently improved that the success of each step can 
be assumed, at least at tolerable cost in redundancy, 
and so are interested in the possibilities of manipulating 
the macrostructure of the process, by continually testing 
the student and utilizing the test result to vary the ma- 
terials presented to the student. Nowhere can the 
essential difference in the two techniques be seen so 
clearly as when we ask the programmer’s purpose in 
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eliciting a response from the student. The linear pro- 
grammer elicits -iJhe response because he believes the 
making of the response is an essential part of the learn- 
ing process. Once the response had been elicited, it 
has served its purpose (in a linear program) and thus 
no further use is made of it. In an intrinsic program, 
the- programmer elicits the response in order to see if 
the student has learned, because this information will 
be used to determine whether the next point is to be 
presented, or whether additional material on the pre- 
vious point is required. Perhaps these considerations 
will make it clear why I am a little discontented when 
I hear someone summarize the field with the statement 
that the' basic difference between a linear program and 
an intrinsic program is that the one uses a constructed 
response, the other a multiple choice response. The 
basic difference is not in the form of the student’s re- 
sponse, bu! rather in the purpose served by requiring 
him to make a response at all. 

A truly automated process is characterized by the 
fact that the result of the process is continuously sam- 
pled and the sample result used to control the process. 
The “feedback” characteristic of an automated process 
is not applied to the work piece, but to the processing 



equipment. I suggest that in this sense, a linear pro- 
gram cannot be considered an automated instructional 
process, since what the student — the work piece, in this 
case — does, does not affect the course of the process. 
An intrinsic program is, of course, an automated proc- 
ess in this sense, as the presentation of material to the 
student is determined directly and continuously by the 
activity of the student himself, and it may be that no 
two students will take the same course through the ma- 
terial. The linear program is an automatic process, in 
that it operates, in principle, untended, but it is not an 
automated process because it is not self-correcting. 

Again, let me say that in drawing this distinction 
between an automatic process and an automated proc- 
ess, I do not intend any unfavorable reflection on the 
simple automatic process. I only wish to make the 
essential difference between the character of the auto- 
matic linear process and the automated intrinsic pro- 
gramming clear. But lest I sound too reasonable, let 
me say that if it takes a fully automated, self-correcting 
process to perform so simple a task as rolling sheet 
steel from fairly uniform steel billets, I do not believe 
that a less flexible process will serve a useful range of 
educational purposes. 



PROGRAMMED INSTRUCTION IN PERSPECTIVE 

Carlton B. Downing 

Carlton Downing points out that the greatest strides in the development and use of 
programmed instruction has token place in the military services. Industry is also 
showing great interest in its use. But in the public schools, progress is moving slowly. 
Producers of programmed materials mode a mistake in programming whole courses 
instead of small sections of courses. Programmed instruction has suffered os much 
from its friends as from its detractors. Better presentation and more empirical 
evidence could speed its acceptance. The by-product of programming, better conven- 
tional teaching. Is one of the most promising aspects of the use of programmed 
materials. 



The subject of programmed instruction in perspective 
can be approached only with some temerity. The title 
implies a global onmicience for which the author’s 
credentials scarcely qualify him. Therefore, this paper 
will limit the discussion to a brief overview of the 
growth, present uses, and the prospects for the use of 
programmed instruction. 

Probably the greatest strides that have been made in 
the development and use of programmed materials for 
teaching have been in the military services. Two of 
the major military centers for the development and use 
of programmed instruction are Randolph and Lack- 
land Air Force Bases of the Air Training Command. 
Programmed instruction within the Air Training Com- 
mand has been favored with sympathetic administra- 
tion, competent leadership, and adequate finances. The 
number of frames written, the number of training skills 
taught with the assistance of programmed materials, 
the number of program writers trained and the amount 
of outside expert technical assistance sought and secured 
support the obvious conclusion that these bases are 
not only leaders in the development of programmed 
instruction for the military services, but that they are 
also actual leaders in the development of the art of 
programming itself. The names of those providing 
technical assistance to the development of programs in 
this area in recent months reads like a roll call of the 
leaders in the art of programming. The effort expended, 
the quality of leadership and level of administrative 
support point to continued vigorous leadership in the 
art of programming in the military services. 

The second area of promise is in the expanded use 
of programmed instruction by business and industry. 
The roster of companies using some phase of pro- 
grammed instruction for training new employees in- 
cludes most of the major corporations of the nation. 
These companies are also using the services of a grow- 
ing number of organizations that provide technical con- 
cultant services in the analysis of skills and the produc- 
tion of programmed materials. 

The National Office of the National Society for 
Programmed Instruction (NSPI) is a vantage point for 



viewing the growing interest of training directors in the 
use of programmed materials. Inquiries have been re- 
ceived from city owned utilities, banks, hospitals, an 
insurance company, a hybrid seed-com producer, and 
a clothing manufacturer. It seems safe to predict from 
these inquiries a broadening interest in the use of pro- 
grammed materials in the smaller companies. The 
acceptance of programs for training in business will be 
determined by one hard and fast business yard stick 
that it must teach the necessary skills more econom- 
ically and more efficiently than other methods. 

The third area to be considered is the use of pro- 
grammed material in the school classrooms. Honesty 
compels the admission that the use of programs by 
teachers has not increased at the pace many would like 
to see. One writer and program producer says ruefully, 
“The schools are enthusiastically NOT using, our ma- 
terials.” Several reasons for this slow acceptance are 
evident. Administrators and teachers are conservative 
because society expects them to be. Historically, par- 
ents have shown considerable reluctance to having their 
children as experimental subjects. It is usually accurate 
to predict that any innovation in teaching will have 
slow acceptance. Innovators in schools have high marks 
for low tenure. 

It appears that early producers of programs for use 
in elementary and secondary schools made a serious 
error in judgment when they chose to program whole 
courses rather than units or modules of courses. The 
decision to teach a semester course with a program im- 
plies the acceptance of the concept of each student 
progressing at his own rate of learning. 

It should have been easy to predict that the use of 
student-paced programs would lead to difficult adminis- 
trative problems. The easily foreseen problem is that 
of what to do with a student who completes the year’s 
course in three months, or even sooner. Principals and 
teachers know what to do for these students, but the 
concept of each student progressing at his own rate may 
demand smaller classes, more teachers, more equipment, 
and materials, and more financial support than most 
communities appear ready to provide. Present also is 
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the fact that no good programs are available for many 
school subjects. Teachers have difficulty in finding time 
to write their their own programs without neglecting 
other duties. 

The cause of programmed material in schools has 
suffered from its friends as much as from its detractors. 
Numerous speakers have been inept and have exhibited 
a vast lack of understanding of the practical aspects of 
teaching. The case for programmed instruction deserves 
better presentation than it has had. Perhaps the single 
greatest deterrent to the acceptance of programmed ma- 
terial for use in the classroom is the dearth of empirical 
evidence to prove its efficacy. 

While the acceptance of programmed material in the 
classroom is not a reason for satisfaction, the picture 
is not all bleak. Some teachers and teams of teachers, 
encouraged by administrators, have found time to write 
and use programs. It seems significant that most 
teachers prefer to write and use modules of courses. 
School counselors are starting to recommend pro- 
grammed material for enrichment for able students and 
for remedial use with students who need it. The use of 
programs in colleges and universities is beginning to 
appear in experimental situations. 

One of the most promising aspects of the use of pro- 
grammed material in classrooms seems to be a by- 
product of programming. But it is one that should be 
heartening to those who believe in using programmed 
materials. Reports recur that teachers who use pro- 
grams begin to apply the techniques to other teaching 
areas. They give more emphasis to stated objectives, 
to pupil involvement, and they assume more respon- 



sibility for the student’s giving correct responses. This 
demonstrates the soundness of one of the fondest claims 
of those enthusiastic about programs, that is, they in- 
corporate the principles of sound and effective learning. 

The forms for writing programs appear to be in a 
changing pattern. Complete adherence to a single type 
of writing frames, be it linear, branching, or other is 
passing. Writers are beginning to use several ap- 
proaches on one program. This practice could well 
lead to more useful programs that embody the best of 
several methods. Another interesting development has 
been the inclusion of adjunct material with programs. 
This inclusion of reading material with frames provides 
extraordinary flexibility for introducing technical and 
unusual material with which the trainee has had little 
or no acquaintance. Writing objectives for technical 
courses is frequently quite simple when compared with 
the problems of determining the starting point in an 
heirarchy of skills. These innovations, this melding of 
methods, gives promise of better programs. 

As a brief summary, the use of programmed material 
has had its greatest use and has its greatest promise of 
growth in the military services. The use of programmed 
material in industry, business, and health services, in- 
creases as it demonstrates its ability to reduce training 
time and costs. The acceptance of programmed mate- 
rial in the vast area of the schools proceeds slowly, but 
acceptance is more wide-spread each year. Better pre- 
sentation and more empirical evidence could speed this 
acceptance. It is encouraging to report that the use of 
programmed material and techniques by teachers re- 
sults in better teaching in other areas. 
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III. Development of Systems Applications 

THE SYSTEMS APPROACH TO PROGRAMMING 

Roger A. Kaufman 

Roger Kaufman points out that the systems approach developed in other industries 
has considcroble value in its application in the preparation of programmed instruction 
materials. The importance of the system approach in education becomes more critical 
with the application of programmed instruction. Programmed instruction is only a part 
or a sub-system of the total educational system which includes school buildings, teacher 
training, and supervision. Roger Kaufman discusses how the systems approach 
achieves better programmed materials. 



The “systems approach” used in many scientific dis- 
ciphnes has important implications for programmed 
instruction. What is the systems approach? Stated 
simply, it is the notion that anything we consider is a 
dynamic whole constituted of interacting factors or 
variables. Perceived in this manner, the whole is really 
more than the sum of its parts. An analogy from 
physiology points out that the human organism is com- 
posed of many parts: structural members, protective 
covering, organs, and integrating and distribution sys- 
tems of various types. Each has a job or jobs, and 
each can be viewed separately and apart from all other 
parts. Yet, in life, the parts do not function alone; we 
know that each part has innumerable interactive effects 
on other parts of the body. The body is not merely a 
collection of parts; it is a dynamic system. All parts of 
a system have interaction effects on other parts. 

Our total world is composed of systems and sub- 
systems. A programmed course in algebra for 9th 
graders, like a Mach 3 aircraft designed to fly from 
New York to Paris in less than three hours carrying 
170 passengers, can be considered a system or a sub- 
system. In a system, each part, or component, has a 
purpose; it contributes in some measurable way to the 
total system objectives. To use the systems approach 
properly, each component must be identified and its 
function determined. If a system has components whose 
existence or functions cannot be identified, the system 
will be relatively ineflScient in meeting its mission ob- 
jectives. Only when all components within a system 
can be identified and their contribution towards achiev- 
ing mission objectives determined can it be said that a 
systems approach is being used. 

In designing a system, using the systems approach, 
we must identify the system objectives, and then con- 
sider all of the possible interactions, or interrelations 
between variables and the ways in which they might 
affect total system performance. 

An automated instructional program may be con- 
sidered a system, or a sub-system within a larger sys- 
tem. If our boundary of consideration is a programmed 



course on map reading, our system is the program, a 
program with specified mission objectives. After the 
course has been written, validated, and implemented 
into a curriculum, the program becomes a sub-system 
within the curriculum system. By using the systems ap- 
proach, persons concerned with creating programs can 
more quickly and efficiently fulfill their “mission ob- 
jectives” of creating valid instructional materials. Actu- 
ally, proper programming requires that a systems 
approach be utilized. Let us now turn to the procedure 
for preparing programmed materials using the systems 
approach. 

Generally, we start with a request for material or a 
stated requirement (see Figure 1).* We analyze the 
requirements for the course, generally performing a 
task and training systems analysis to determine exactly 
what skills and knowledges the trainee (student) must 
have when he completes the course and the character- 
istics of the training matrix into which the programmed 
course will be introduced. We analyze the student 
population to determine what behavioral repertoire the 
students will bring to the course. From the task analysis 
and student population characteristic data, we set the 
objectives for the course, objectives which are measur- 
able. From these objectives we derive a criterion test 
which can be used to determine when we have satisfied 
the mission objectives, or the extent to which our pro- 
gram has not been completely successful. The next 
step is to obtain the approval of the training manager 
or curriculum director since he is responsible for the 
programmed system when it becomes a sub-system in 
his total training system. If the training manager is not 
satisfied with the derived mission objectives and/or 
criterion test, they are revised. After the training man- 
ager’s approval, we determine what training methods 
and techniques are available for performing our mission 
objectives. Using the data collected from the task 



* This figure is a slight variation of one originally developed by 
the Training Methods Division of the Air Training Com- 
mand, USAF. 
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analysis in conjunction with the objectives, we deter- 
mine our strategy for creating the programmed course. 
The first program draft is then written to meet the 
previously determined system objectives. It is reviewed 
for technical accuracy and for clarity and communica- 
tion skill. It is then rewritten and tried on several tar- 
get population students. This tryout is known as frame 
or item validation and supplies data for a program 
revision based on student response data, not conjecture 
by the programmer or other personnel concerned with 
the program-system. Frequently, the training manager 
is asked to review and accept the programmed materials 
after frame validation and revisions. After approval, 
the program is subjected to a group tryout. This tryout 
is a “dress rehearsal” which determines if the program 
meets its objectives. Generally, a large number of stu- 
dents from the system target population are given a 
pretest, the program, and then the criterion test. The 
difference between the pretest performance and the 
post test performance yields an index of content gain. 
Based on group performance on the program as meas- 
used by the gain scores, the program is revised appro- 
priately. After final approval by the training manager, 
the program is assembled in final form and put into use. 

Ordinarily, one might consider that our systems task 
has been completed, but such is not the case with pro- 
grammed instruction. We must continue to collect stu- 
dent response data to improve the program, to enable 
us to meet our mission objectives more completely or 
efficiently. The programming system, then, is a closed 
loop since we constantly feed information back into 
the system to improve system performance. 

In using the systems approach, we have proceeded 
from a set of specified objectives to the creating of a 
system, and have done so in an orderly, planned, and 
controlled manner. We have taken account of all the 
factors which might affect system performance. We 
have also taken into account possible interactions be- 
tween factors. In using the systems approach to pro- 
gramming, we have created a model for system-build- 
ing. This model should be open to revision as wc 
collect additional data on how to create valid programs. 
It is a science of programming. 

The systems approach is really the formalization of 
proven techniques for accomplishing complex tasks 
efficiently. A model for progressing from the specifica- 
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tiou of mission objectives to the satisfying of those 
objectives in a planned manner is economically as well 
as philosophically sound. In programming instruction, 
the systems approach is valuable not only as a guide 
for the creation of programs but as a model for the 
appropriate utilization and integration of programs into 
a total training situation. 

Other models for program creation have been sug- 
gested. Skinner has offered one based on conditioned 
responses. Crowder has offered a model based on hu- 
man communication interaction. Unfortunately, these 
models have usually been unable to predict the out- 
comes of experimental research concerned with the 
operating characteristics of various program formats. 
These models, in their current form, presently seem in- 
adequate for forming guidelines for preparation of 
programmed material. 

The systems approach offers an alternative to in- 
adequate theory, or at least provides a supplement until 
such time as theory matures to the point of providing 
usable criteria for programming. The systems approach 
evokes the criteria of functional utility; we select tech- 
niques and methods in terms of specific desired educa- 
tional or training outcomes. We thus shift from a 
technique orientation to a problem orienh tion. 

The importance of systems approach to training and 
education has been forcefully enhanced by the require- 
ments of programmed instruction. It might well give 
rise to an educational technology which will surpass 
and enhance programmed instruction’s usefulness for 
improving education and training. 

By utilizing the systems approach, includmg the stat- 
ing of educational objectives and missions in measur- 
able behavioral terms, and by planning the implementa- 
tion of change througjh accounting for and manipulation 
of variables contributing to system objectives, education 
and training can become an efficient, sensible, orderly 
endeavor. In the same manner that we constuct a 
program using the systems approach, we can also plan 
curriculum units, school units, school buildings, and 
entire educational systems. Educational and training 
objectives can and should be quantified. We need to 
start planning our educational systems around measur- 
able behavioral outcomes, and the systems approach 
seems to be an excellent model for getting under way. 
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PROGRAMMED INSTRUaiON AS A SYSTEMS APPROACH TO EDUCATION 

Robert E. Corrigan 

Robert Corrigan proposes that programmed instruction be conceived of as a formal 
instructional system best suited and designed to meet the ultimate objectives of our 
educational system. System requirements are postulated to organize and develop the 
methods and materials of Instruction including automatic teaching most consistent with 
efficient individual learning requirements In both individual and group settings. He 
further proposes that the underlying philosophy of this system provides the most mean- 
ingful rationale to organize, coordinate and direct the efforts of all contributing 
groups. Applying the proposed systems concept provides a “team approach" with 
specification of the role and performance requirements for each team member con- 
sistent with the established objectives of instruction. 



During the past three years, programmed instruc- 
tional methods have been extensively explored and 
evaluated at all levels within our national educational 
system. The results substantiate the positive signifi- 
cance of these methods for education. While educators 
acknowledge the significance of self-instructional pro- 
gramming methods, they are posing penetrating 
questions concerning the application of programmed 
instruction to the broader scope of instructional re- 
quirements which face them. A specific realism ex- 
pressed by the educator is the requirement for both 
individual and group instruction. Dr. J. Lloyd Trump, 
National Education Association, presents a plan of 
action which recommends that 60% of the average 
30-hour week be utilized for large and small group 
instruction, and 40% for individual instruction. His 
report, and others, emphasize the group or class as the 
basic instructional unit in our educational system, at 
least, for the foreseeable future. Our concern in group 
instruction is that each individual within the group at- 
tain his greatest potential level of achievement by our 
providing the most appropriate instructional and indi- 
vidual learning conditions to insure this. 

Educators are asking these questions: (1) how do we 
meet and satisfy the needs for group instruction?, (2) 
what is the role of the programmed instruction to meet 
broad curriculum directives?, and (3) how do we com- 
bine the elements of programmed instruction in terms 
of meeting the realisms for a mass educational system? 
I propose that programmed instruction be conceived of 
as a complete system of instruction, best organized and 
designed to meet the specific objective of efficient in- 
dividual learning. I believe that the methods of pro- 
grammed instruction when applied to the design of 
instructional systems provide the means for resolving 
the broad scope of instructional requirements which 
include the need for both individual and group instruc- 
tion. 

The terms “system” and “systems approach”, when 
originally conceived, related to applied methods for 
increasing the efficiency of overall planning, organiza- 



tion, and coordination in the development of our 
national weapon systems. The “systems approach” 
considers each of the many individuals and groups 
developing a particular weapon (like the Atlas mis- 
sile) as individual components like cogs in a machine 
working together to achieve a common goal. This ap- 
proach requires: (1) defining in advance the job of 
each person or “cog” in terms of his assigned respon- 
sibilities, (2) detailing the task, (3) specifying per- 
formance requirements, and (4) stating the necessary 
interactions and communications to be carried out be- 
tween groups, each requirement tailored and defined 
to meet the pre-set system goal. The word system, 
therefore, suggests an organized plan carried out in de- 
tail to achieve a pre-established goal or objective. The 
goal must be accomplished within set performance 
limits. Quality of performance is measured by the 
degree of compatibility or efficiency between those 
components making up the system in the attainment of 
pre-established terminal performance objectives. 

Now let us apply the systems approach to pro- 
grammed instruction. Here we are concerned with one 
common objective, namely, efficient individual learning 
and performance. In meeting this objective, we can 
define three scopes of activity which are performed by 
independent groups. (Table I.) These groups and as- 
sociated activities are as follows: (1) the curriculum 
director and the programmer analyze and develop in- 
structional materials; (2) the instructional environment 
consisting of the teacher-student-program complex, 
carrying out the methods and procedures in the in- 
structional process as established by the curriculum 
director and programmer; and, the administration, 
supervisory, and parental groups, who, while operating 
outside the immediate instructional environment, di- 
rectly, or indirectly, influence the quality of instruction 
and individual learning. Each will be discussed 
separately. 

The systems model is the contribution of the human 
factors or human engineering analyst (Table II). He 
applies the systems approach: (1) to establish design 
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T = Teacher 
S = Student 
P = Program 



Table I — Programmed Insiructional System 
Components 



specifications for equipment, and (2) to establish op- 
erating and training requirements for the man operating 
the equipment. He must describe and predict perform- 
ance for the man, design requirements for the machine. 
This method is called “task analysis.” His prime ob- 
jective is to insure that man and machine are com- 
patible as a unified system. 

In an instructional system the “teacher-student-pro- 
gram complex” replaces the “roan-machine” system. 
The programming analyst using the task analysis model 
follows a continual “try and fit” procedure to “tailor” 
a specific program or instructional package to meet 
defined instructional and learning objectives. The prob- 
lems confronting the programming analyst cover: (1) 
the curriculum directives and instructional require- 
ments, (2) the learning requirements of the students, 
(3) the characteristics of the leamer(s), (4) the in- 
structional setting, and (5) the models and methods of 
communication. All these the programming analyst 
must consider while developing instructional materials. 

Individual achievement is the direct outcome of the 
interaction betv/een the three prime sources which 
make up the immediate teaching-learning environment. 

TEACHER 

STUDENT-< ^-PROGRAM 

Each of the three elements (teacher, student, and 
progrcm) represent the contributing elements necessary 
to the instructional setting. While each can be con- 
sidered alone, we define the teaching-learning process 
as a dynamic event with each element continuously in- 





SYSTEM MODEL 

1 

SYSTEM OBJECTIVES 

ADVANCE PLANNING 
ASSIGN FUNCTIONS 




Table II — Programmed Instruction as a Systems Model 

teracting with the others in an ever dependent relation- 
ship. The efiiciency of teaching and/or learning is the 
direct outcome of the quality of the contribution of 
each of the three defined elements at the moment of 
their interaction. This is relevant whether we are re- 
ferring to a student and a program, a teacher and a 
student, or any combination of these. 

In analyzing instructional and learning processes we 
must consider all contributing elements influencing sys- 
tem efficiency. Our attention must be directed to the 
broadest scope to include groups outside the immediate 
instructional environment. Viewing our educational 
system as an iceberg, the tip is representative of the 
immediate classroom. As with an iceberg, the greater 
supporting groups represented by administrative, super- 
visory and parental groups, although non-observable, 
directly influence the quality and efficiency of instruc- 
tion. It is imperative, therefore, that the role and con- 
tribution of each support group be defined to provide 
peak efiiciency. Applying the common objective, stu- 
dent achievement, the required contribution of each 
can be analyzed and integrated into a comprehensive 
plan with means established for measuring the quality 
and efficiency of performance. The underlying guiding 
principle of the systems approach to education provides 
this vehicle. 

Independent of system components or groups are the 
requirements and scope of the proposed instructional 
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system. I should like to propose eight requirements 
of this system.* These are as follows: (Table III). 



DESIGN 

REQUIREMENTS 


DESIGN 

CLASSIFICATION 


1. Statement of objectives 




2. Essential subject matter 


y ORGANIZATION 


3. Order material in steps ^ 




4. Active student partici- 




pation 


COMMUNICATION 


5. Controlled pacing 


and 


6. Two-way communica- 


CONTROL 


tion 

J 




7. Apply learning princi- 




pies 


EVALUATION 


8. Performance require- 




ments-measures 





Table ill — Design Requirements ond Classifications 
for a Programmed instructional System of Education 

1. The statement of objectives for instruction and 
individual learning expressed in performance terms. 

2. The determination of the essential (minimal) 
subject matter to meet the stated objectives. 

3. The ordering of subject matter into a program 
format designed to expedite the learning route. 

4. Highly individualized student participation on a 
continuously active basis featuring the recurrent re- 
quirement for overt, observable responses to strengthen 
the learning process and to evaluate student achieve- 
ment. 

5. Controlled pacing of instruction consistent with 
student performance as measured by the pre-established 
criteria of learning. 

6. Highly directed communication by the “instruc- 
tor” using the “tutorial” or “coach-pupU” two-way 
communication model to insure continual and purpose- 
ful interaction between “instructor and student”. 

7. Incorporation throughout the program of those 
fundamental principles essential to efficient learning, 
such as: 

a. Knowledge of results of performance. 

b. Immediate correction of incorrect responses. 

c. Purposeful repetition and reinforcement schedul- 
ing. 

d. Directing the student in purposeful learning se- 
quences with his prior knowledge of learning objectives 
present and future. 



♦Expressed broadly in a policy statement by USAF, Hq., 
Washington, D.C. The stated requirements as originally pre- 
sented have been modified and expanded. 
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8. Statement of interim and final performance re- 
quirements and measures. 

The design characteristics of the programmed system 
fall into three major classifications, organization, com- 
munication and control, and evaluation. 

Organization would deal with the underlying philos- 
ophy directing the system such as the analysis and 
planning for individual and group learning needs; and, 
the selection of methods to design materials to meet 
these defined needs. We would concern ourselves first 
with what is needed, second with which strategies of 
program construction should be used, and then con- 
sider how to complete the analysis and step sequences 
in the writing phases for either individual and/or group 
applications. 

Communication and control is conceived of as pri- 
marily the teaching-learning process. Within this major 
classification is subsumed (1) highly individualized 
active student participation, (2) the methods and 
model of communication employed, and (3) those 
factors affecting the pace and efficiency of the learning 
process. 

Evaluation of programmed materials would be 
guided by the performance criteria specified during the 
planning and analysis phases. We would be concerned 
with the meeting of these performance (learning 
achievement) criteria in terms of final or termmai stu- 
dent achievement and with program standardization 
during the validation phase. Within this framework 
we would consider (1) the underlying philosophy in the 
determination of performance measures, (2) the ways 
and means of specifying these measures, and (3) the 
meaning and use of terminal performance measures as 
final achievement tests. 

The eight propositions or requirements provide a 
broad framework for an efficient system without re- 
stricting ways and means of implementation. Regarding 
the scope of such a system, I make four assumptions 
(Table IV) : First, the application of the principles of 
programmed instruction is not restricted to self-paced 
or self-instructional materials, but can be extended to 
group instructional methods as long as the require- 
ments are rigorously met. Second, automated teaching 
or the use of teaching machines is not necessarily a 
contingency of programmed instruction. The discipline 
of programmed instruction can be applied advantage- 
ously to any educational or training situation and does 
not necessitate the automated or teaching machine 
process. However, within certain situations which both 
permit and warrant it, the process of automated teach- 
ing can be achieved. Thus, the two terms, programmed 
instruction and automated teaching are not, as com- 
monly mistaken, synonymous. Programmed instruction 
is the system, while the use of teaching machines for 
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self or group paced applications must be examined in 
terms of unique contribution to the efficiency of learn- 
ing, instruction, or both, for the specific application. A 
third assumption is that mass media of communication, 
per se, are not necessarily eliminated. When we use 
such media as closed or open circuit television, motion 
pictures, tapes, film strips, etc., we are transmitting in- 
formation. If these media are modified to rigorously 
meet the stated system requirements, they can be used 
to teach each individual within the group on a highly 
individual basis. An important fourth assumption is 
that the proposed system does not limit methods or 
strategies of programming which now include linear, 
branching and mathetics. Each and all can be used 
advantageously and make unique contributions within 
the system. Neither environmental setting nor instruc- 
tional application is limited, providing that all the 
requisite criteria are met. 

These stated postulates can be applied to our exist- 
ing instructional environment to accomplish optimal 
design of instructional sequences. In fact, there are 
significant by-products to be derived from partial ap- 
plications of the proprosed system. For example, imme- 
diate benefits can be derived by applying the stated 
organizational principles appropriate to programmed 
instructional methods for lesson planning procedures 
(Table V). Lesson planning, as employed now, or- 
ganizes the teacher in terms of subject matter coverage 
and serves primarily as a guide. In contrast, I propose 
that the lesson plan should be considered as a student 
learning blueprint with the prime design measure ex- 
pressed in terms of student achievement. To be truly 
meaningful, teaching objectives should be defined in 
terms of specific accomplishment, not subject matter 
coverage. Further, accomplishments should be defined 
in terms of student accomplishment, not teacher ac- 
complishment. Applying the systems method, the 
teacher would establish specific student learning objec- 
tives, pre-state the methods and measures of student 
performance, and organize teaching steps to meet the 
learning objectives. You might comment that what I 
am suggesting is that the teacher is simply teaching the 
final examination. That is exactly right! With pro- 
grammed instructional methods, however, the final ex- 
amination does not represent a sampling of the concepts 
taught (as does conventional testing), but, rather, 

* SELF PACED 

* GROUP PACED 

A. CLASSROOM 

B. MASS MEDIA 

* ALL PROGRAM STRATEGIES 

* TEACHING EQUIPMENT OPTIONAL . 

Table IV — Scope of Programmed Instruction System 



demonstrates performance by the students of all pre- 
established achievement measures covering the com- 
plete and total defined criteria of learning. 

Let us examine the proposed concept that teaching 
and learning be considered primarily as a communica- 
tion and control process. In the conventional group 
setting, the teacher most often applies the lecture or 
“one-way” communication model. Students play a pas- 
sive role. The teacher proceeds according to his best 
“guestimate” of how well the information is under- 
stood. But who knows what each student has learned? 
For group instruction to be truly effective, the lecture 
method must be replaced by the tutorial or “two-way” 
communication model. In this context, communication 
is defined as including three processes: (1) sending, 
(2) receiving, and (3) understanding. Without the 
assurance that there is “understanding” of that which 
has been “sent” and “received”, we have not commu- 
nicated. 



CONVENTIONAL 

1. Teacher oriented 

2. Subject coverage 

3. Teacher guide 

4. Instruction objectives 

5. Test samples 



PROGRAMMED 

1. Student oriented 

2. Performance steps 

3. Learning blueprint 

4. Learning objectives 

5. Test total measure 



Table V — Comparison of Lesson Planning Methods 



I am not proposing merely more question-answer in- 
teraction between teacher and student, but purposeful 
controlled questioning designed to test pre-set learning 
criteria measures. Lowry AFB evaluated the extensive 
use of classroom question-answer techniques allowing 
the students the freedom to stop the instructor during a 
presentation to ask questions. The results of this study 
(Table VI) indicated the following: 60% of the ques- 
tions asked by students anticipated the instructor’s 
presentation; 30% of the questions were completely 
irrelevant and seriously disrupted the instructional se- 
quence; only 10% of the questions were meaningful 
and significant. In summary, 90% of student questions 
disrupted the flow and pace of instruction without con- 
tributing to the individual learning process. 

Teaching groups using the tutorial model of com- 
munication with programmed instruction implies that 
the program proceeds in a group-paced manner. 

The obvious questions to this proposal are: (1) how 
does this take care of the problems of individual differ- 
ences, and (2) how does group instruction using pro- 
grammed methods affect the individual’s rate of 
learning. 

Charles Frye, at Oregon College of Education, work- 
ing under an N.D.E.A. grant, investigated ways to 
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STUDENT QUESTIONS AT ANY TIME 

Q * 60% ♦ ANTICIPATING 

U 

E 

S 

T 

I ♦ 30% ♦ IRRELEVANT 

O 

N 

S ♦ 10% ♦ MEANINGFUL 

90% Disrapt without contributing to individual 

learning 

Table VI — Lowry Question Study 

adapt self-instructional techniques to a group setting 
(Table VII). (Frye 1958) Frye assigned individuals 
to groups according to similiarities for such character- 
istics as ability and experience in order to learn a linear 
program in mathematics. For group instruction, mate- 
rial was presented to experimental groups on slides in 
a group-paced manner. The groups proceeded at a rate 
established by the slowest learner. He compared 
group-paced versus self-paced methods for the same 
material. The results indicate that students of “like” 
or homogeneous ability perform as well working under 
group-paced versus self-paced methods for the same 
their own pace. Frye concludes that “only when there 
is a wide discrepancy in abilities within a group is there 
any likelihood that learning rate will be retarded by 
forcing the learners to progress at the rate of the 
group.” 

A form of communication model proposed for group 
instruction which meets all the stated system require- 
ments for programmed instruction is the classroom stu- 
dent response communicator such as the “Monitor- 
Teletest Communication.” This system provides the 
capability for a teacher to program materials in care- 
fully designed teaching steps according to the specifica- 



GROUP VS. INDIVIDUAL PACING IN 
PROGRAMMED INSTRUCTION 



Median Time In Minutes To Complete 



Pacing 


Homo- 


Hetero- 


Difference 




geneous 


geneous 




Group 


255 


325 


70 


Individual 


221 


240 


19 




34 


85* 





* Statistically significant (P. 01) 

Table VII — Oregon College of Education 
(Charles H. Frye) 

tions stated. As an integral part of the teaching, criteria 
questions are posed to the class. Each student responds 
privately, is given immediate knowledge of the correct- 
ness of his response and has the ability to select the 
correct answer, if in error, before the group proceeds 
to the next question. As the students respond, the 
teacher is given immediately the percentage of correct 
answers for the students’ first response. In addition, 
a continuous record of each student’s performance can 
be presented. The Monitor-Teletest Communications 
System has been tried extensively at San Jose College, 
University of Southern California, Alamitos School 
District, Garden Grove, California and at the USAF 
Academy. In the Air Force Academy program, two- 
way co mmuni cation was extended to four classrooms 
over closed circuit television. Instructions for all class- 
rooms was directed by one television instructor. 

In every case, the results show strikingly superior 
gains in the efficiency of instruction and individual 
learning when compared with conventional group in- 
structional methods. I would like to present, in digest 
form, the results of three research programs using the 
Monitor-Teletest System with group-paced program- 
ming methods in classroom instructions. 
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DIGEST I 

PLACE: San Jose State College, San Jose, California 
INVESTIGATOR: Arthur Vicory 

PROJECT: The roll of student feedback (knowledge of results) in group-paced programming. 

PROGRAM: “Developmg of Number Systems” natmral through complex numbers. 

POPULATION: 87 college students 

STANDARDIZATION: Step I: 15 individuals to 95% correct response on program 

Step II: 3 students in group, high, medium, and low mathematics aptitude. 
EXPERIMENTAL DESIGN: Three experimental; one treatment group. 

Experimental: Treatment 

45 minutes of Instruction 250 Minutes of Instruction 

Groups: 1. This group did not represent an experimental con- 

1. Student response with feedback » Heatment group to be used in 

comparison of conventional and programmed in- 

A. Monitor-Teletest System structiou methods. 

2. Student response — no feedback 

A. Marked answers on IBM answer sheet. f .?>“Sht simto materials 

m a formal course m mathematics, rive days of 

3. No student response instruction were required to cover materials pre- 

A. Conventional instruction sented with the 45 minute programmed sequences. 

Criterion test: 43 questions measuring recall and problem solving given one week following instruction. 
RESEARCH RESULTS: 

1. Monitor-Teletest Group (with student response and immediate feedback) performed significantly better than 
experimental groups (a) responding without immediate feedback and (b) taught conventionally. 

2. There was no significant difference between Group 2 (student responding but without immediate feedback) and 
Group 3 (conventional instruction). 

3. Treatment group (tau^t in mathematics course) required 250 minutes of instruction to perform 8% higher 
than group with feedback and 45 minutes of programmed instruction. 

1. Passive student response without immediate knowledge of results on a private basis has no 
significant effect on learning. (Groups 2 and 3). 

2. Active student response with immediate knowledge of results reinforces learning and in- 
careases performance beyond limits of mathematics program, per se. (Group 1). 

Table VIII— 'CoFitclusions (Vicory) 
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DIGEST II 

PLACE: University of Southern California. Los Angeles. 

INVESTIGATORS: William Deutsch and Harold Gluth. 

PROJECT: Comparison of group-paced programmed instruction with student feedback and response versus con- 
ventional instruction (no programming, no student response). 

PROGRAM: Principles and use of Projection Systems. 

POPULATION: 65 college students. 

EXPERIMENTAL DESIGN: 

Two control groups: 1. Two-hour instruction period. 

A. One hour lecture, one-half hour demonstration, one-half hour laboratory. 

2. No student responding or immediate feedback. 

Experimental Group Experimental Controls 



1. One hour group-paced program covering identical 
material. 

2. Student responding with immediate feedback with 
classroom student-response communication sys- 
tem. (Moaitor-Teletest). 



1. Control instructors shown in advance program- 
ming objectives to insure same instructional goals 
both control and experimental group instruction. 

2. Instructors qualified and experienced. 



Criterion Test: Standardized 76 True-False examination given two days after instruction. 

1. Experimental group using Monitor-Teletest System performed 25% higher in achievement 
tests than control groups. 

2. No differences reported between control groups. 

3. 50% savings in time of instruction (one l^our) using group-paced programmed methods 
with student responding with feedback. 

Table IX — Research Results (Deutsch) 



DIGEST III 

PLACE: Alamitos Intermediate School, Garden Grove, California. 

INVESTIGATORS: Dan Kaufman, Betty Corrigan, and R. E. Corrigan. 

PROJECT: A comparison of group tutorial instructional methods for science instruction with seventh grade 
students. 

PROGRAM: Human Skeletal Systems. 

POPULATION: 186 seventh grade students. 

EXPERIMENTAL DESIGN: 

Group Tutorial Methods Investigated 
Method I Method II 



1. Group programmed instructional sequence on Hu- 
man Skeletal Systems taught using overhead pro- 
jector system, but without classroom student re- 
sponse system. (Teacher-paced. No direct student 
feedback.) 

2. Qualified science instructor. (One of team prepar- 
ing programmed sequence.) 

3. Five days of instruction follwed by terminal achieve- 
ment test. 

4. Retention test: 16 days later on identical examina- 
tion. 



1. Identical group programmed instruction materi- 
als on Human Skeletal Systems taught using over- 
head projector system but with Monitor-Teletest 
System. (Teletest-paced. With individual student 
response and immediate feedback.) 

2. Qualified science instructor. (One of team prepar- 
ing programmed sequence.) 

3. Four days of instruction followed by terminal 
achievement test. 

4. Retention test: 16 days later on identical examina- 
tion. 



Criterion Test: 100 questions covering multiple choice (28), true-false (25), matching (25), and recall of techni- 
cal data (22) designed in advance of instruction to test all pre-established terminal student performance 
(learning) objectives. 
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EXPERIMENTAL RESULTS 

1. Group progran?.med materials increased the scope, 
depth, and complexity of instructional coverage by 
a factor of 5:1 as compared with this unit taught 
conventionally. Comparisons between instructional 
material specifications were made by the science 
instructors involved in the experiment. 

2. Group programmed materials were taught in 5 
days as compared with a 9 day instructional period 
previously required — a reduction of 60% in time 
of instruction for programmed methods. 

3. The criterion examination for the programmed in- 
structional sequence consisted of 100 questions. The 
prior examination for the conventionally tau^t 
unit consisted of 48 questions — an increased ratio 
of 2: 1 for length of exam with the programmed se- 
quence. The difficulty of the examination was re- 
flected in the ratio of 5:1 increased coverage for 
the programmed sequence. 

4. Qualifying students taught with instructional Meth- 
od I (teacher-paced but without direct student- 
teacher interaction) achieved an 80% mean correct 
response on the initial test with a mean percentage 
correct of 73% on a retention test presented 16 
days later interim period — Christmas Holidays). 
A relative loss of only 7% in student mean per- 
formance was experienced following retest. 

5. Qualifying students taught with instructional Meth- 
od II (the programmed sequence using the class- 
room student teacher response system Monitor- 
Teletest) achieved a mean of 84% correct responses 
on the initial test with a mean percentage correct 
of 78% on the retention test presented 16 days later 
(interim period — Christmas Holidays). A relative 
loss of only 6% in student mean performance was 
experienced following retest. 

SUMMARY 

1. Using group programmed materials (teacher- 
paced), the efficiency of instruction — as compared 
with a comparable unit taught conventionally — ^was 
raised by a factor of 10:1 as measured by scope, 
depth, complexity and time of instruction; and 
provided a 12% increase in mean achievement 
scores (80% versus 68% ) with a mean error per- 
centage loss in retention of 7% over a period of 16 
days. 

2. The combining of the Monitor-Teletest Communi- 
cation system (M'^thod II) with the programmed 
sequence provided a 22% increase in mean per- 
formance on the initial test and retention test as 
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compared with teacher-paced instruction (Method 
I) using group programmed instruction. 

With the Monitor-Teletest Communication system, 
group instruction becomes a hi^y individualized 
learning experience for the student and the teacher. 

San Jose State College 

Possible Score = 100 % Correct 

(During Prog.) Criteria Questions 

Group 1 (Student feedback) N = 24 

Mean 85.83 66.11 Students Respond 

2 

S.D. 52.71 26.21 

Group 2 (No student feedback) N = 22 Programming 

Mean 80.64 62.27 

2 

S.D. 64.01 42.16 

Group 3 (No student responding) N = 20 

Mean 61.21 Conventional 

2 

S.D. 64.80 Lecture 

Group 4 (No student responding) N = 21 Methods 

Mean (Treatment 72.57 Group 3 — 45 minutes 
2 Group) Group 4 — One week 

S.D. 1 15.56 in course (250 min- 

utes). 

Table VIII — Research Results — Monitor-Teletest 

The United States Air Force Academy has developed 
a complete instructional system combining the ele- 
ments of self-paced and group-paced programmed 
instruction. Group-paced instruction is accomplished 
by combining the Monitor-Teletest Syston with closed 
circuit television (mass communication). A two-way 
communication network is provided between students 
in four classrooms and the television teacher in the 
studio. The system gives the teacher immediate knowl- 
edge of the level of understanding indicated by students 
and the rate at which problem solving is accomplished. 
With this knowledge, the teacher controls the level of 
complexity of instruction and the pace consistent with 
student needs. Students in turn are provided immedi- 
ate feedback and reinforcement. As used in this appli- 
cation, the Monitor-Teletest System becomes a 
teacher-paced system. Following the televised instruc- 
tion, students work on self-paced materials followed 
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by practical problem solving tasks applying concepts 
learned. This systems model of programmed instruction 
should be significantly interesting to educators. The 
results have been of great significance to the Academy. 

University of Southern California 

* STUDENTS RESPONDING WITH FEEDBACK 

25% increased test achievement over both con- 
trol groups (P .01) 

* 50% SAVINGS IN INSTRUCTION TIME 
Using group-paced programming with Monitor- 
Teletest System 

* NO DIFFERENCES BETWEEN CONTROLS 

Table IX — Research Results — Monitor-Teletest 

One of the questions posed by educators is: What 
is the role of the self-instructional program(s) in 
broad curriculum directives? Combining self-instruc- 
tional materials in the school system has taken other 
forms than applying individual instruction in place of 
group methods. Two studies in particular, Teaching 
Machines, Inc. and AIR, indicate the significant 
advance in individual learning and achievement when 
self-paced programs are used in homework assign- 
ments. Significant contributions are reported in over- 
all individual learning efficiency and student motiva- 
tion when self-instructional programs are integrated 
into the instructional system in this and other manners. 

The Relevancy of Programmed Instruction 
as a Systems Approach to Education 

In March, 1963, a two-week meeting sponsored by 
the United Nations was held at the UNESCO house*, 
Paris, France attended by educational representatives 
from the majority of major participating countries in 
the world. The subject of these meetings was: “Nev/ 
Methods and Techniques In Education.” The scope of 
discussions included a wide range of problems relat- 
ing to support, research and development, and imple- 
mentation of educational systems and applications. 

\\fith reference to methods and media planning, spe- 
cific attention was directed to (a) radio and television, 
(b) electronic computers, and (c) programmed instruc- 
tion methods and applications. The key to system 
implementation was integration of methods and media 
for more efficient learning and teaching efficiency. 

Information presented in this appendix represents 
verbatim statements included in the UNESCO Report. 

♦Report of a meeting of experts: New Methods and Tech- 
niques in Education UNESCO/ED/ 190, Paris, France 



Several points, among others, were presented ex- 
pressing group concurrence concerning the nature of 
methods and media. These are: 

A. Programmed instruction is to be considered as a 
significant metliod calculated to enhance the value of 
instructional media. 

B. The teacher remained the focal point for educa- 
tional work. 

C. Stress was laid on the need for “combining the 
best elements” of traditional teaching with judicious 
use of the nev/ techniques. 

D. Techniques (media) were becoming increasingly 
adaptable to integration in various educational sys- 
tems. 

E. Electronic computers might constitute an instru- 
ment for the efficient management of education (the 
establishment of the optimum use of time, premises, 
and staff; measurement of pupil progress; and, in 
general, a dynamic view of educational phenomena) . 

The expert group further concurred on the following 
points defined as educational needs and psychological 
aspects: 

A. Educational needs, both in and out of school, 
were becoming steadily greater. 

B. Teaching and learning efficiency is increased: 

(1) by giving pupils fuller and more accurate infor- 
mation about the matter under study, with a resulting 
improvement in the quality of their knowledge; 

(2) by enabling pupils, especially the better ones, to 
assimilate more, and more quickly; 

(3) by making it possible to evaluate that assimila- 
tion more accurately and more objectively; 

(4) by preparing the fundamental algorithms for the 
teaching of basic subjects. 

C. Interaction methods should be used for the im- 
provement and development of all educational materi- 
als (films, television programs, material for self-teach- 
ing purposes, and, indeed, all types of audio-visual aids 
or printed media). In particular, individual or collec- 
tive “feedback” techniques should be used at every 
point for testing and revision, in accordance with the 
usual self-instruction procedures. 

The group further concurred on the implications of 
these new media and methods for educational systems. 

A. The use of such new techniques, especially tele- 
vision and programmed instruction, tended to affect 
the size of the groups being taught and the rate at 
which individual students or groups learned, and to 
add to the number of methods of presentation. 

B. The systematic use of such methods and media 
gives rise to new problems of organization, teacher 
training, specialization of technical staff, production, 
distribution, and publications. 
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C. Such proposed changes are bound to lead to pro- 
found changes in school organizations, rearrangement 
of premises, coordination of teacher tasks and the re- 
arrangement of educational practices (individual exer- 
cises, small group discussions, collective presentation 
to large groups, either lineally or with feedback). 

D. The fundamental responsibilities of the teacher’s 
function will not be modified, or diminish the vital im- 
portance of human realtions in teaching. 

E. To develop improved methods of assessing the 
comparative costs of parameters which can be ad- 
justed to differences in cultures, their requirements 
and their local resources. 

Based on these stated conclusions, the international 
educational expert group recommends through UNES- 
CO that, among others, the following recommendations 
serve as a basis of action for instructional system plan- 
ning. 

A. Particular attention is recommended to the de- 
velopment and use of imaginative ways of combining, 
for maximum educational gain at minimum costs, the 
resources of mass media, or self-instructional pro- 
gramming methods, and of teacher teams (instruc- 
tional systems planning). 

B. Special emphasis should be placed on adapting, 
to all media, the techniques of feedback from individual 
students of audience groups of mass media. Such feed- 
back to the producer should consider factors of accep- 
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tance and attitude as well as instructional efficiency 
(group classroom and television student two-way re- 
sponse systems). 

C. Research of a fundamental nature is needed to 
improve basic understanding of the learning process 
(programmed instruction application) . 

D. Special emphasis should be placed on developing 
the potential of individual programmed instruction 
methods (self-instruction and group-instruction). 

E. An intensive effort should be devoted to obtaining 
and collecting data on comparative monetary costs 
and expenditures required for alternative means that 
seem to be capable of attaining a particular kind of 
needed educational outcome. Factors taken into con- 
sideration in collecting such data should, however, not 
only be the monetary outlay, but also the effect on 
manpower, on the required time, and on the improve- 
ment of quality of instruction (advanced derivation of 
parametric operating data for instructional systems 
planning) . 

The conclusions and recommendations of this inter- 
national group of educational experts points out in 
clear language the required ways and means for future 
plLiining. The emphasis on instructional system plan- 
ning combining programmed self-instructional and 
group tutorial applications strongly supports the signi- 
ficance of this proposed systems approach to educa- 
tion. 
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PROGRAMMED INSTRUaiON AND THE TECHNOLOGY OF TRAINING 

Robert G. Smith, .fr. 

Robert Smith presents a concept of a technology which concerns the use of seven 
steps to be followed in the development or revision of training programs: 1) Analysis 
of both operational and training systems, 2) Analysis of fob, 3) Analysis of tasks, 
4) Determination of training objectives, 5) Construction of training programs, 6) Devel- 
opment of proficiency measures, 7) Evaluation of training program. 



In the nearly twelve years of HumRRO’s existence, it 
has done research which has made a significant impact 
on Army training procedures. 

In one of our earliest studies, for example, we have 
revolutionized training in Rifie Marksmanship, so that 
today’s soldier learns to shoot in a realistic, combat- 
like situation, rather than in an artificial situation in- 
volving white cards with black dots on them. 

We have been able to reduce training time in elec- 
tronics maintenance specialties by as much as one- 
half, without loss of proficiency. 

The successes we have had in reducing training time 
and increasing proficiency are due to the development 
of a concept of the Technology of Training. By this 
we mean a group of seven steps, which if taken in 
sequence, repeatedly create improvement in training. 

These steps are: (1) Analysis of the operational sys- 
tem and the training system; (2) Analysis of the job; 
(3) Analysis of the Tasks to specify required knowl- 
edges and skills; (4) Determination of training ob- 
jectives; (5) Construction of the training program; 
(6) Development of measures of proficiency; and (7) 
Evaluation of the training program. 

You are probably saying to yourself — ^This looks all 
right as a model for conventional instruction, but what 
does this have to do with programmed instruction? 

My theme is that it has everything to do with pro- 
grammed instruction. Following these steps is the best 
way to avoid those failures of programming which 
appear in the literature these days. By failure, I mean 
cases in which programmed instruction does only as 
well, or not as well, as conventional instruction. 

The first four steps are devoted to making a precise 
determination of what is to be tau^t. Historically, 
HumRRO has made its greatest successes in this 
area. By studying the system in which the soldier is 
to work, by analyzing the job, eliminating the nice to 
know, and teaching what the soldier must know, we 
have consistently obtained significant results. 

These findings are strongly supported by Gagne 
(1962a), in his presidential address to the Division of 
Military Psychology of the American Psychological 
Association. Gagne said: “Here are the psychological 
principles that seem to me to be useful in training: 
(1) Any human task may be analyzed into a set of 



component tasks which are quite distinct from each 
other in terms of the experimental operations needed 
to produce them; (2) These task components are medi- 
ators of the final task performance; that is, their 
presence insures positive transfer to a final perform- 
ance, and their absence reduces such transfer to near 
zero; and (3) The basic- principles of training design 
consist of: (a) identifying the component tasks of a 
final performance; (b) insuring that each of these com- 
ponent tasks is fully achieved; and (c) arranging the 
total learning situation in a sequence which will insure 
optimal mediational effects from one component to 
another.” 

Gagne (1962b) further states that the traditional 
principles of learning, such as reinforcement, differenti- 
ation of task elements, familiarity, and distribution of 
practice are much less important in training than the 
principles of task analysis, intra-task transfer, compo- 
nent task achievement, and sequencing. 

Now let me discuss the seven steps and their rele- 
vance to programming. First we, analyze both the op- 
erational system and the training system. A system 
is a group of inter-related and inter-communicating 
components, organized to achieve a goal. Some of the 
components may be machines, others will be people. 
Systems also contain rules or policies which affect 
their operation. They are also embeded in larger sys- 
tems, which place constraints upon the smaller sys- 
tems. 

The student is a part of a training system. When he 
graduates he will become part of an operational sys- 
tem. 

By analyzing the operational system we discover the 
context of the job he is to perform. We learn of policies 
which restrict him to the use of certain kinds of tasks, 
and eliminate the performance of other tasks, which 
we might think it nice to teach him. We learn of his 
role in contributing to the mission of the system. We 
learn which of his tasks are vital to the mission of the 
system, and which are less essential. This helps us 
determine the level of proficiency we will require him 
to reach. 

By analyzing the training system we discover what 
we should expect the student to bring to our particular 
course, and the possibility of later skill development, 
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perhaps on-the-job training in another course. We 
leam of policies which, rightly or wrongly, may place 
constraints on the kinds of training we may ^ve him. 

Step 2 is an analysis of the job. Many training people 
tend to think that there is just one kind of job analysis. 
Actually there are several kinds, and for several dif- 
ferent purposes. The kind of job analysis performed 
for setting up personnel classification systems, or job 
evaluation, just won’t do for training. Th^ are loo 
general. 

A job analysis for training must identify all the ma- 
jor duties in the job and the tasks which make up these 
duties. The physical and environmental conditions of 
the work must be described. We must specify the tools, 
the test equipment, the job aids and manuals the worker 
must use. Finally we must determine the standards he 
must meet, in terms of accuracy and time. 

At this stage we have a list of the duties and tasks 
which must be performed. We must now proceed to 
step 3, the specification of knowledges and skills. 

The initial step is to analyze each task to be per- 
formed in stimulus-response detail. We must know 
what must be done, how fast, and how accurately. We 
must identify the signals or cues which tell when an 
action is to be performed. We must describe the action 
precisely, and we must describe the feedback which 
tells whether the action has been performed correctly. 

One of the reasons for making a precise det ;ription 
of a task is to identify the kind of symbolic material 
needed to perform the task. The symbols are usually 
called knowledge. The precise determination of knowl- 
edge is extremely important. One of the most common 
failiugs, and greatest wastes in training, is to teach 
more knowledge than we have to. There are large 
bodies of knowledge, theory and history surrounding 
nearly every human endeavor. But what may be nec- 
essary for &e scientist or engineer in a given field may 
be not at all required by the technician. 

Certainly there will be specific knowledges required 
in any job. Among these will be: (1) nomenclature; 
(2) location of work or components; (3) identification 
of tools, etc.; and (4) specific work rules and proce- 
dures. 

In some instances a general principle enables a man 
to respond to situations he has never precisely en- 
countered before, or at least to reduce his trial and 
error in making the correct responses. But these prin- 
ciples are ivory-tower unless they are anchored in the 
real word. 

Let me give an example: We have a bit of knowl- 
edge, in the form of an idea, statement, or self instruc- 
tion. This might be, “Never ground a hot electrical 
conductor”. 



Robert G. Smith, Jr. 

First, we need the perception of things in the real 
world which require this knowledge. For example, the 
trainee must perceive bare wires, terminals, junctions, 
prongs, and other objects as potential hot electric con- 
ductors. He must leam to perceive what a grounding 
condition is, but he must leam these perceptions as part 
of the knowledge. 

The student must also leam the real world responses 
indicated by the knowledge. He can put the knowledge 
to work by not standing in water, or by using a mb- 
ber or bakelite shield, insulated tools, or mbber gloves, 
or by avoiding physical contact with the object he per- 
ceives as a hot electric conductor. 

Unless a principle has been anchored on both the per- 
ceptual side and the response side in the actual work 
situation, we can never be sure that the knowledge is 
really necessary. 

This is important. Unless we follow it, we will be 
teaching whole courses in mathematics when the job 
requires only the ability to make a few simple compu- 
tations. Or, we will be teaching the theory of equip- 
ment operation instead of the job the worker has to do 
with the equipment. 

The importance of this idea is reinforced by the work 
of Shriver (1960) (Shriver, Fink and Trexler, 1961) 
in HumRRO research Task FORECAST. In the 
field of electronics trouble shooting, it has been tradi- 
tional to teach the theory of electronics. This includes 
a body of general theory, as well as the theory of op- 
eration of particular circuits in particular equipment. 
From this knowledge the repairman is supposed to de- 
termine where ihe malfunctioning component is located. 
In a system which may contain over 100,000 compo- 
nents, this is no mean trick. Using his theoretical 
training, the repairmen is expected to dete rmin e where 
to make checks so as to narrow his search, and he is 
expected to compute the electronic values he is expecte<l 
to find at that point. 

Studies of the behavior of repairmen show that this 
system doesn’t work in practice. They make more 
checks than they need to. They check in the wrong 
places. Frequently they do not use test equipment 
properly. 

The conventional theory taught is that required to 
design the equipment. The significance of Shrivefs 
work is that he developed a new theory of electronics 
systems, a theory designed from the maintenance 
point of view. This theory involves a restructuring of 
the system into trouble-shooting blocks, which are dif- 
ferent from usual electronic chassis packaging. In ad- 
dition, a simply learned trouble shooting strategy was 
taught. This strategy was: (1) from symptom to symp- 
tom area; (2) from symptom area to trouble-shooting 
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block; (3) from block to tube chain; and (4) from tube 
chain to individual part. 

What was the result? The result was repairmen 
trained in 400 hours who were as proficient as repair- 
men trained in the conventional course of 1000 hours. 
And in another study, a small amount of additional 
practice resulted in a significant increase in profi- 
ciency. 

Step 4 is the Determination of Training Objectives. 
There are two matters of significance here. First, we 
must select what we are to teach in our particular 
course or program. An objective is the statement of 
the behavior we expect the student to display at the 
end of our program. This is often short of the profi- 
ciency ultimately desired on the job, depending on 
time, facilities, cost, and the opportunity for later on- 
the-job training. 

Several factors should be considered when we select 
what to teach, and the level of proficiency desired as 
a result of our program: 

1. What does the student know already? We seldom 
take a close look at what the student brings to our 
course. Yet there may be significant amounts of what 
is required on the job already known by the student. 
This is especially true when, for example, already 
trained technicians are retrained in new equipment. 

2. What will the student learn in later courses, or on 
the job? We should not permit unplanned redundancy 
or duplication to exist. 

3. What job aids are available on the job? Many as- 
pects of job performance are supported by check lists, 
manuals, tables of data, slide rules, etc. The avail- 
ability of job aids should be considered when deciding 
on objectives. K a man will look up data in a manual, 
he doesn’t need to learn how to derive this data. Or, if 
he will use a check list, there is no need to require him 
to memorize the sequence of actions in a procedure. 

4. What must the student be able to do immediately 
on the job, vs. the things he can develop on the job? 

5. What behaviors are critical to the performance of 
the operational system? There are many tasks or parts 
of tasks which are not very critical. There may be a 
supervisor who can remind the worker if he fails to 
perform correctly. Yet there are other tasks on which 
failure may mean disaster, either to the worker or 
the system. 

Having determined what to teach, we must describe 
bur objectives in behavioral terras in suflScient clarity 
to communicate to our students and others in unam- 
biguous terms. I know of no better treatment of the 
problem of describing objectives clearly than Mager’s 
(1961b), clever and amusing scrambled book. Prepar- 
ing Objectives for Programmed Instruction. 



I have devoted a considerable part of presentation to 
the matter of determining precisely what to teach. 
Why all this concern over precise, job related objec- 
tives? The reason is simply that it is quite costly to 
teach the wrong things, either conventionally or via 
programmed instruction. Let’s examine some of these 
costs. 

First, the costs of teaching too little are fairly obvi- 
ous: (1) Additional training on the job will be required. 
There are variations in job situations, but often the job 
is arranged for work, rather than for training. There 
may be less time and poorer facilities for teaching on 
the job than in school. In addition, there may be com- 
plaints from the users of your graduates which can be 
embanassing; and (2) Personnel may be endangered, 
or equipment damaged. Production my be slowed. 

A much more subtle cost is teaching too much. This 
is more subtle because you will not hear complaints 
from the users of your graduates. Yet the costs are 
there just the same: (1) Teaching more than is needed 
to do the job will require longer courses, more instruc- 
tors, and more programs than are necessary; (2) 
Students may fail on the irrelevant parts of the course, 
even though they may be capable of perfo rmin g well 
on the job; and (3) Irrelevant criteria may be used to 
select students to attend the course. Irrelevant pre- 
requisites may be established. Then students who 
could learn the job don’t get a chance to do so. 

It is for these reasons that I have come to believe that 
the problems of deriving clear, job related objectives 
are the most important in training. 

Step 5 is the Construction of a Training Program. 
For Programming, this is the equivalent of frame 
writing. I am not going to talk about the art of writing 
frames. This has been described too often to be re- 
peated here. Instead, I would like to mention some re- 
search findings which I feel would, if put to use in a 
programming context, make significant contributions 
to success. 

Several years ago, HumRRO conducted a study un- 
der the code name RADAJR. VI (Hitchcock, Mager and 
Whipple, 1958). This study resulted in a reduction of 
training time from 33 to 26 weeks for electronic main- 
tenance technicians. At the same time, the graduates 
of this course were as proficient as men who had 18 
months of field experience, a significant improvement 
over the control course. One of the things which was 
done in RADAR VI was to give copies of the course 
objectives to the students. So the student’s time could 
be spent in meeting the objectives rather than in crys- 
tal-balling, or in military slang “G-Zing”, what the in- 
structor wanted. This can save a tremendous amount 
of time. 
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Those of us who have taught can remember the ques- 
tions the students ask us to try to discover our objec- 
tives. “Will this be on the exam? Do you give multi- 
ple choice or essay tests? Do we have to know how to 
derive this formula or just how to use it?” These are 
some typical questions. 

Those who have been students recently can remem- 
ber the time we spent talking to former students of our 
courses to find out v/hat the professor really wanted. 
Among graduate students coming up for qualifying 
examinations this can reach a high art. 

Why don’t we give the students our objectives? 
Whether the course is programmed or not, this can 
be quite effective. It will remind the student of our 
goals when he gets bogged down in the detailed frames 
of a program, wondering where it all leads to. 

When you are pre-testing, or de-bugging your pro- 
gram, giving the student the objectives for your pro- 
gram can be useful. The student can then raise ques- 
tions about the relevance of portions of your program 
to your objectives. Our currrent procedures for pre- 
testing programs are biased in favor of detecting places 
where we have gone too fast for the student. The re- 
visions we make tend to add frames and lengthen the 
program. We need corrective measures to eliminate 
some of our pet topics which may not be relevant to 
our objectives. It seems to me that giving the try-out 
student the objectives of the program may provide this 
necessary counter-balance. 

One of the problems faced by the programmer is 
that of sequencing. In a long program, dealing with 
many topics, arranging those topic' in an effective 
sequence will be one of the prog j mer’s toughest 
problems. 

HumRRO research has provided a sequencing prin- 
ciple which should be useful in making the broad deci- 
sions about sequencing topics. This has been termed the 
Functional Context Principle (Brown, Zaynor, Bern- 
stein, and Shoemaker, 1959). One of the more inter- 
esting aspects of this principle is that it seems to work 
especially well for students of lower aptitude (Goffard, 
Heimstra, Beecroft and Openshaw, 1960). 

In Functional Context sequencing, we always main- 
tain a job-related context for the material we present. 
Translating this into educational terms, we might be- 
gin by describing how certain knowledges might be 
used in real life. We might begin with an overview of 
the job itself. In RADAR VI we tau^t maintenance 
men how to operate the equipment at the very first 
part of the course. By this means, the students learned 
the purpose of the equipment, how it looked and felt. 
There are many ways by which we can give a man a 
general view of his job. 



In detail, the Functional Context approach ha'- 
following features: (1) Instruction is aimed at proQue- 
ing effective maintenance men, without regard to com- 
petence in electronic theory. The theoretical material 
presented was only that which had clear cut applica- 
tion to the repairman’s job; (2) All skills and knowl- 
edges were taught in a whole-to-part or trouble-shoot- 
ing sequence. Thus, early in the course, system trouble 
shooting was taught, followed by work on smaller stages 
and, finally, the isolation of components; and (3) 
Each lesson is designed to provide a framework, or 
context, for the next one. 

Thus, from the overview of the job we become more 
and more specific. Each step we take is related to the 
job context previously established. The student learns 
new skills or knowledges only when they are required 
by the job. 

The sequence then moves from the whole to the part. 
In electronics maintenance, at least, this has the inter- 
esting result that basic electronics, which deals with 
components, gets taught last, rather than first. 

We are all used to being taught in the seemingly logi- 
cal way of learning “fundamentals” in the absence of 
any tie to the purpose of learning these things. But 
this is not a very efficient way. 

By using the Functional Context principle in laying 
out the general sequence of our programs we should 
be more successful in making them more meaningful 
and interesting to the student. 

But what about more detailed sequencing problems? 
You will remember that I stressed earlier the position 
of Gagne that we need to analyze tasks into compo- 
nents which mediate transfer to final task perform- 
ance. Gagne (1962b) has recently made suggestions 
for making this analysis. Starting with a final task, 
he asks, “What would the individual have to be able 
to do in order that he can attain successful perform- 
ance on this task, provided he is given only instruc- 
tions? This question is then asked of the subordinate 
tasks or classes of tasks, until we can go no further. 

Gagne has shown that students who have not mas- 
tered the subordinate tasks cannot achieve the higher 
order tasks. Further, the way to achieve mastery of 
the final task is not to practice the final task alone but 
to learn the subordinate tasks which lead up to the final 
task. 

The combination of these ideas, the Functional Con- 
text notion for the broad sequence, and Gagne’s anal- 
ysis for the more detailed sequencing, should make 
the programmer’s sequencing decisions easier. 

Step No. 6 is the Development of Measures of Pro- 
ficiency. Proficiency tests are scorable samples of 
actual job performance. HumRRO has found perform- 
ance tests more appropriate for determining training 
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progress than written tests. We want men who can ac- 
tually do the job, not just talk about it. 

V/e have developed electronic trouble-shooting tests 
which require the actual location of faulty components 
in real equipment. We have required soldiers to navi- 
gate on land from one point to another, and not just 
answer questions about using the map and compass. 

When preparing a program designed to teach an 
actual job task, it will be necessary to develop a per- 
formance test to be sure our programs have actually 
taught this task. 

Step No. 7, our last one, is the Evaluation of the 
Training Program. Proficiency tests must be routinely 
used in order to insure control over quality of the train- 
ing program. The results of testing must be fed back 
to the instructor or programmer so that he can make 
appropriate changes in the program. The tests should 
also be used to determine the effect of the changes 
made. 

Just what is the significance of the seven steps just 
described for programming? 

We have all been impressed by the cases which 
showed that programs resulted in learning with sav- 
ings of time of 25-50% over conventional instruction. I 



would like to point out that HumRRO, by following the 
Technology of Training has achieved comparable sav- 
ings without the use of programs. How much more ef- 
fective would programs be which were based on these 
concepts? 

Another reason for stressing the Technology as ap- 
plied to programming is to reduce the risk of a pro- 
gramming failure. Raab(1961) provides a case in point. 
Raab evaluated a set of mathematics programs pre- 
pared for junior high school students. These pro- 
grams did not teach as well as conventional instruction 
and took longer. The students complained that they 
were bored and did not know where they were headed. 
A review of Raab’s report from the standpoint of the 
Technology is enlightening: (1) The program included 
material the students had learned before; (2) No ob- 
jectives were provided for the student; (3) The tests 
used to evaluate the programs were not selected or 
prepared on the basis of the program; and (4) The 
programs did not provide a basis of meaningful con- 
text for the student. 

In summary, let me say that the notions of the Tech- 
nology of Training provide powerful concepts for any 
kind of training, programmed or not. 



SEHING PROGRAMMED INSTRUCTION OBJECTIVES USING SYSTEMS METHODOLOGY 

Charles S. Morrill 

Charles S. Morrill has presented a clear seven step procedure for setting objectives 
into a systems framework using systems methodology. The advantage of using systems 
methodology In setting objectives for programmed instruction is to increase the prob- 
ability that Ae resulting programs will be an integral part of the training system. Some 
less obvious advantages may prove equally interesting to many readers. 



Clearly, one of the major objections in using the 
word "systems” in the title of this paper is that much 
controversy exists concerning an appropriate defini- 
tion for this word. The fundamental principle, for ex- 
ample, of systems design, has been stated as follows: 

“Maximize, in some sense, the expected value.” 
(Goode &Machol, 1957) 

Recently Bamford (1962) has written a short treat- 
ise entitled On the Meaning of “System”. In this 
paper he attempts to define formally the concept of a 
system by interpreting and formalizing some proper- 
ties ascribed to systems and to construct a language 
for the disciplined discussion of systems. Others (El- 
lis & Ludwig, 1962) have tried to define systems in 
terms of precise mathematics. In order to study and 
define the functions of a specific system, Gordon 
(1962) states that, “First, a model of the system to be 
studied must be constructed, and then a program that 
embodies the logic and action of the model must be 
written”. He goes on to say that, “The model is a set 
of equations and logical statements that expresses the 
relationship between the components of the system and 
the various parameters associated with the system”. 

While it is true that dissension is rampant concern- 
ing the concept of a “System”, there is a core element 
within this dissention which is applicable to using sys- 
tems methodology in setting objectives for program- 
med instruction. The objective, then, of this paper 
is to explain one approach to accomplish this end. 

In reviewing the work done in personnel selection, 
Dudek (1963) has stated that, “Several papers brought 
out the fact that the study of personnel selection and 
classification in relation to such things as assignment, 
placement, manpower utilization, training, group struc- 
ture, man-machine systems, and organizational man- 
agement is more meaningful and profitable than the 
study of personnel selection in vacuo”. Although Du- 
dek’s emphasis is in the area of personnel selection 
and classification, he very well makes the point that 
training is a part of a system. The concept of a train- 
ing system includes all of the facets which are men- 
tioned above. In discussing selection problems, Haire 
(1959) emphasized the need for adequate criteria, and 
similarly noted that “. . . the assessment of training 
continues to be much discussed but is dealt with only to 



a limited extent”. Haire’s position is that coordina- 
tion between related areas where identification and 
development of adequate criteria are concerned is es- 
sential. 

Let us assume, for the moment, that programmed 
instraction is a method of teaching and training which 
is part of an overall training system. Our first step, 
then, in developing a program is to prepare objectives 
for that particular program. Mager’s recent book 
(1961b) in this specific area outlines these procedures 
most effectively. And it is true that if Mager’s proce- 
dures are followed, the objectives thus derived will be 
clear, specific, and measurable. However, it is neces- 
sary to look beyond the particular objectives for the 
program to insure that these objectives are appropri- 
ate to accomplish the overall training or educational 
goals. Permit me to outline seven steps for setting 
objectives using systems methodology: (1) Examine 
overall objectives of the educational or training sys- 
tem (as defined before) in which the progranuned ma- 
terial will be used; (2) Determine which overall ob- 
jectives of the system would be most efficiently and 
effectively achieved using programmed instruction; 
(3) Develop the objectives for the programmed mate- 
rial and check to determine if they are in any way in 
conflict with the overall objectives of the system. K 
no conflict, fine. If conflict exists, ask such questions 
as: 

a) Are the program objectives achievable in view of 
the constraints imposed on the overall system, e.g. 
money, time? 

b) Are any specific program objectives directly in 
conflict with those of the system? 

c) Do the program objectives help to fulfiU the ob- 
jectives established for the system? 

This procedure checks the program objectives early 
enough in the development procedure to prevent a pro- 
liferation of inconsistencies; (4) Look at the entire sys- 
tem to determine if other methods of instruction (either 
being used, or which could be developed) would ful- 
fill the objectives for the programmed material and the 
training objectives of the system more effectively and 
efficiently; (5) Given that the program objectives are 
appropriate, examine the system and the environment 
in which it operates to see if there are any indications 
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that there is a best method of PRESENTATION for 
the programmed material. For example, are there any 
constraints, e.g., money, space, which would obviate the 
use of certain kinds of teaching machines? Is there 
a machine already chosen for which the programmed 
material must be written?; (6) Survey the field of 
available programmed material to determine if mate- 
rial already available or in the process of being devel- 
oped might well fulfill the stated objectives. Clearly, 
in some areas which are specifically related to a par- 
ticular company operation, progutms will not be 
available; and (7) If programmed material is avail- 
able, determine the quality of the material, and how it 
will satisfy the objectives which have been established. 
Although already developed materials may not f ulfill 
all the established objectives, it may be worthwhile to 
modify either the objectives or, if possible, the pro- 
gram, rather than develop new programmed material. 

While it may be true that the above seven steps will 
not guarantee that programmed material will be an in- 
tegral part of a training system, the probability is 
greater, it seems to me, that the resulting materials 
will prove more useful as part of an educational or 
training system than if the steps are not followed. The 
essential point, however, is a simple one. Program- 
med material either developed in isolation or thought 
of as a unit of material unto itself with “no strings at- 
tached” will probably be less effective than materi- 



als developed in a systems framework using systems 
methodology. 

The use of systems methodology in establishing ob- 
jectives helps bring together several talented people, 
many of whom will represent different technologies. 
The fact that these interested and qualified people are 
uniting their efforts to determine objectives in and by 
itself will help to renew the vigor and raise the stand- 
ard for development of both specific programs and 
training and educational systems. Secondly, a review 
of the entire system in terms of objectives may very 
well highhght previous errors in both materials and 
stated goals. Training and educational systems are 
dynamic. Thus a review occasioned by the addition of 
programs to an already existing system may have a 
catalytic effect on the entire system. Thirdly, addi- 
tional cost which must be incurred to program mate- 
rial, which may already be part of a curriculum, pro- 
vides a stimulus for close scrutiny of the entire 
teaching or training system. Often the mere introduc- 
tion of programs will force changes in overall sys- 
tem objectives and/or procedures. The last obvious 
advantage is that an analysis of progiam objectives 
in terms of total system objectives forces an evaluation 
of both. It is important indeed to stimulate interest in 
the evaluation of system objectives as a focal point 
for the design of subsystems based on programmed 
materials. 



THE IMPLEMENTATION OF A SYSTEMS APPROACH TO 
PROGRAMMED INSTRUCTION FOR A STATE DEPARTMENT OF EDUCATION 

William F. Ryan 

William F. Ryan concurs that our greatest barrier in moving into a new plan for 
improving State education departments will be our loyalty to the inadequate organi- 
zations and arrangements which exist today. If we can recognize them as being the 
best we could imagine when we invented them, give them full credit for what they 
have accomplished, and then move on to something better. States can maintain 
leadership in education. 



The organization for education on the state level is 
little different structurally from other organizations 
such as any business, industry or the military. All 
across America experiments with new concepts and 
practices are beginning to show significant results in 
the improvement of quality. Many states have taken 
the lead in this movement at both the state and local 
levels. These efforts, however, are not enough. They 
need to consider how best to bring to bear the total 
educational resources of the state in a massive attack 
on resistance to constructive change. Recommenda- 
tions for statewide action have been undertaken to 
evaluate new practices and devices, to initiate and ex- 
pand constructive experimentation in our schools and 
to facilitate and accelerate widespread use of prac- 
tices and devices which have proven successful in their 
schools. Specifically, the high administrators have 
listed in their programs of study educational television, 
team teaching, large and small group instruction, un- 
graded classes and teaching machines and program- 
med learning. 

The State of New York, for example, with reference 
to the Brikell report, approached the systems con- 
cept toward newer instructional media with certain 
natural self-imposed limitations. 

First, they did not investigate classroom practice 
due to the extremely wide variance of behavior of indi- 
vidual teachers with their classes. Second, they did not 
assess educational institutions of higher learning from 
any aspects other than their impact on instructional 
innovation in local public school systems. 

How does one go about conducting a study? The 
following methods have been used to conduct a study 
on innovations in state school systems: schools were 
selected to find out how they made decisions about in- 
structional change, not to observe or to assess their 
new programs. One state had a project which for sev- 
eral years involved collecting elaborate data from a 
sampling of over 300 different school systems which 
represented the total range of types and sizes of the 
state. 

Active, well-financed suburban school systems were 
chosen somewhat more often than other types because 



it seemed important to study closely those conditions 
which are most conducive to rapid educational change, 
according to previous research into community types. 
The fact that such school systems were generously 
represented in the survey gives a special mgency to 
the conclusions in this report. If the conditions which 
affeec change even in well-supported, suburban schools 
are as described here, then in other school systems not 
so richly endowed with resources, the pace of change 
is presumably even slower, the direction even less 
certain. 

The State of New York was used for a pilot survey 
by the Center for Programmed Instruction and the 
U. S. Office of Education in determining the variance of 
usage, attitudes and program reorganization change 
from 1962-63 to 1963-64 programs. The study in- 
cluded the following components: (It should be noted 
that the word “program” refers to “programmed learn- 
ing materials.”) 

1. Parents and citizens groups in most communities 
do not exert a direct influence on the adoption of new 
types of instructional programs, but their influence is 
decisive when exerted. In disseminating new pro- 
grams, it is not necessary to arouse the active enthusi- 
asm of local parents, but it is necessary to avoid their 
active opposition. 

2. The board of education in most communities is 
not a strong agent in determining the oath of educa- 
tional innovaton, but its influences are decisive w’hen 
exerted. New programs must be disseminated in a 
manner which will not arouse the opposition of boards 
of education. 

3. New types of instructional programs are intro- 
duced by administrators. Contrary to general opinion, 
teachers are not change agents for instructional inno- 
vations of major scope. To disseminate new types of 
instructional programs, it will be necessary to convince 
administrators of their value. 

4. Qassroom teachers can make only three types 
of instructional change in the absence of adminstra- 
tive initiative: 

a. Change in classroom practice. 

b. Relocation of existing curriculum content. 
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c. Introduction of single special courses at the high 
school level. 

Classroom teachers cannot be ejspected to introduce 
new types of instructional programs without adminis- 
trative attention and in-service courses designed for in- 
dividual teachers rather than for entire departments or 
faculties (that is, courses designed to improve the 
teacher rather than to change the program) should 
be limited to matters which can be accomplished in 
one of the three ways mdicated above. 

A teacher who returns to his school with new infor- 
mation and new teaching skills is most likely to use 
them if he can employ them in his own classroom with- 
out distrubing the work of other teachers. If his new 
information and skills demand the creation of a new 
course for their realization, he is most likely to get 
administrative permission for a course at the end of a 
sequence which will cause least disturbance to the work 
of other teachers. 

1. Professional suspicion about the value of innova- 
tions in other school systems, and even about the sin- 
cerity of other innovators, is a widespread and serious 
inhibitor of educational change. Recommended new 
programs must be disseminated in a way which will 
allow the practitioner to quell his own suspicions. 

2. The most persuasive experience school personnel 
can have is to visit locations where successful new 
programs are in action. Speeches, literature, research 
reports and conversations with participants outside 
the actual instructional setting are interesting but 
relatively xmconvincing. Recommended new pro- 
grams must be demonstrated so that they can be ob- 
served in action. 

3. Anything abnormal, unreal, or artificial in the cir- 
cumstances surrounding an observed program, that is 
to say, anything appreciably different from conditions 
in the visitor’s own school system, can rob a visit 
of persuasive effect. Recommended new programs 
must be demonstrated in schools quite similar to those 
from which visitors come. 

4. New instructional programs can be successfully 
introduced despite initial apathy or even opposition 
on the part of a number of teachers. 

5. The m.ost successful innovations are those which 
are accompanied by the most elaborate help to teach- 
ers as they begin to provide the new instruction. In- 
structional innovation should be accompanied by 
substantial, continuing assistance to teachers. 

6. Instructional innovations are many times evalu- 
ated by observing the reactions of the students. In the 
eyes of the practitioner, no other evidence out-weighs 
student reaction as a measure of success. More com- 



plex evaluative techniques are rarely used. Observ- 
ers who visit demonstrations of recommended new 
programs should be given an opportunity to talk with 
students about their reactions to the new instruction. 

7. Instructional innovations usually seem to the peo- 
ple using them to be distinctly better than what they 
were doing before. The reactions of users cannot be 
relied upon as the sole method of evaluating novel ap- 
proaches. The attention, encouragement and recog- 
nition given to teachers by people outside the class- 
room during the introduction of new programs are 
among the strongest causes of their success. In eval- 
uating new programs, attention, encouragement and 
recognition should be supplied equally to both the ex- 
perimental and the comparison settings. 

A state education department exerts a powerful 
influence on educational change in local schools. In 
some ways the department is an active stimulus; in 
other ways it is a serious barrier. It soon became 
obvious during the survey that the elementary and 
secondary schools look to their state departments for 
leadership and feel much more comfortable in making 
instructional innovations when the department has en- 
dorsed them. Thus the posture of the department to- 
ward innovation in general and toward innovation in 
particular is of great import. 

Following recommendations for a coordinated state- 
wide effort in the State of New York, we propose that 
state education departments do the following: 

1. Create a semi-autonomous agency to stimulate 
and to finance the design and rigorous evaluation of 
new instructional programs for elementary and secon- 
dary schools. 

2. Create field locations all across the state to achieve 
the enriched settings needed for the design of new' in- 
structional programs. School and college people 
should be hired temporarily, brought together and 
freed to design a program. 

3. Design programs to be put through an elaborate 
statewide field test to find what they will accomplish. 
Here again, the evaluators should be agency staff 
members of schools and colleges hired temporarily 
to do the job. 

A state educational plan to introduce progr ammin g 
should have three functions: (1) to demonstrate proven 
and endorsed programs, (2) to teach teachers in the 
component districts how to carry out the new pro- 
grams, .and (3) to arrange for nearby colleges and 
universities to teach the necessary skills on a continu- 
ing basis. New programs should be demonstrated by 
regular teachers in regular schools. The necessary 
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new skills should be taught by a staff of the talented 
school and college people hired temporarily for that 
purpose. 

A number of existing organizations in State Depart- 
ments are now performing operations which would be 
of far more significance to the schools if they were 
paic of an overall plan for accelerating the pace and 
improving the direction of educational change. 



William F. Ryan 

The work of the state education department, the 
colleges and universities, the lay and professional as- 
sociations, the philanthropic foundations, and the com- 
mercial organizations should be coordinated with the 
woik of a state education department. 

Some other existing organizations are performing 
operations which could be better performed under an 
entirely new arrangement. 



BEHAVIORAL PLANNING NETWORKS 



Gustave J. Rath 

Gustave Rath presents a technique for interconnecting terminal behaviors to be 
taught in an auto-instructionai program. Four quasi-mafhematical rules are used to 
organize the behaviors into an efficient planning network of minimum size. The 
finished net can be used to guide programmers toward the most efficient sequence 
of frames. It also guides the program editor in making meaningful but integrated 
assignments to members of the pr^'gramming team. 



General agreement exists upon few things in the 
field of programmed instruction. However, everyone 
agrees that one of the main requirements, and most 
critical jobs, in the development of programmed mate- 
rials is the establishment of proper objectives in the 
form of terminal behaviors. A behavioral planning 
network allows one to organize the terminal behaviors 
according to the real structure of the material. Thus, 
this network aids in the planning and execution of the 
program. 

A behavioral planning network is a technique for 
describing the real structure of the knowledge taught 
by a self-instructional program. Auto-instructional 
programs or programmed instruction materials con- 
sist of sequences of material which modify the behavior 
of the learner in a more or less permanent fashion. 
The modification of the behavior is effected through 
the use of psychological techniques and art which have 
been extensively described in the literature. 

Definitions 

A programmed sequence consists of a series of psy- 
chologically and factually sound teaching events which 
are prepared through the use of adequate materials 
which take an organism described by an existing state 
of measurable behavior and bring it to a new state 
of measurable behavior. This new state might be 
achieved by the addition of a new behavior, the elimi- 
nation of an old behavior, or the modification of an 
existing behavior. A programmed sequence is rep- 
resented by an arrow as shown in Figure 1. 

The number appearing under the arrow indicates the 
number of steps this program sequence is from the be- 
ginning of the network (described by the beginning 
terminal behavior). 

A terminal behavior (TB) is represented by a cir- 
cle with incoming arrows and departing arrows rep- 
resenting programmed sequences which precede the 
development of the terminal behavior and show the di- 
rection to the next step. The graphical representation 
of a TB is shown in Figure 1. Each terminal behavior 
must be named and described in measurable, opera- 
tional terms. For the purposes of symbolism, they 
may be numbered, but each behavior must refer to 




Programmed Sequence — S is the 
a number of steps from the begin- 
ning terminal behavior 

a Terminal Behavior = K is a 
part behavior such that 




K = N 

H *K = i 

K= 1 

a Ending Terminal Behavior — has 
no programmed sequences fol- 
lowing it. 

a Beginning Terminal Behavior — 
has no sequences leading to it. 



Figure 1. Definitions 

a real measurable entity. A lower-case letter repre- 
sents complete terminal behaviors. 

A beginning behavior is a terminal behavior which 
requires no antecedent programmed sequences for 
the population being trained. The specification and 
analysis of the population begins with the existing be- 
haviors, and these are the beginning behaviors of the 
network. -i\n ending terminal behavior, which is rep- 
resented by an upper case letter in a circle, has no 
succeeding programmed sequences. The purpose of a 
program is to establish a set of specified behaviors 
in the student. 

It is understood that a TB may consist of several 
part behaviors of the desired task, and in many cases 
different programmed sequences are necessary to 
separately elicit the part behaviors. In some cases, 
the synthesis of two part behaviors into a terminal be- 
havior occurs simply by the sequential non-ordered 
application of two programmed sequences; that is, 
one may apply one and then the other with no regard 
to order. Having both sequences presented, the full 
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behavior may then be elicitied. Part behavior is sym- 
bolized by letters, such Ik. A terminal behavior is 
made up of the aggregation of several part behaviors, 
K = N 




K= 1 



Ik, as seen in Figure 1. When simple ag- 



gregation cannot occur by juxtaposition, then intermedi- 
ate aggregating sequences are necessary. Rules for 
sequencing are presented below: 



Rules of Aggregation 

Given a sequence, as illustrated in Figure 2, which 
has as its ending terminal behavior the beginning be- 
havior of another sequence, the two may be aggregated 
or combined to form a larger sequence. 




Figure 2. Rules of Aggregation 
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Figure 3. Rules of Synthesis 



behavior which may be followed by two or more termi- 
nal behaviors independently, these must be shown as 
emanating from it as their permanent beginning be- 
havior. Minimum antecedents may be operationally 
defined by using the following formula. If one takes 
Sm, a sum of the number of steps on each programmed 
sequence from its beginning terminal event, one may 
M = N 

use an index D S-m. The sum of squares 

M = 1 

must be minimized to achieve the needed end. Figure 
4 shows that the more sequences that can emanate 
from the beginning or early behaviors, the shorter the 
networks that will be developed. This is not to imply 
that a program will be written and presented as de- 
scribed by the network according to the curriculum 
described later. The description of the relationships 
of the terminal behavior in this manner is very im- 
portant and useful. 



It is possible to have a network where some sequences 
will never be aggregated. For instance, a course di- 
rected toward teaching students how to drive a car 
may have a completely independent sequence on how 
to buy license plates. In many cases, procedural 
sequences are not organically related to the technical 
core and, therefore, do not have to be shown as related 
parts of one network. 

Rules of Synthesis 

Given two sequences which are unrelated, but which 
have sequences which terminate in related part termi- 
nal behaviors, one may synthesize the two sequences 
by forming a new terminal behavior which is a combi- 
nation of the two existing part behaviors. 

The rule of synthesis is illustrated in Figure 3. The 
synthesis presumes that the part behaviors be additive 
and do not require additional steps to achieve it. In 
those cases where additional steps are required, it is 
not a synthesis but an aggregation. 

Rules of Minimal Antecedents 

The rule of minimal antecedents is directed toward 
shortening the networks. If there exists a beginning 




A PROGRAM SHOULD BE ARRANGED SO 

M = N 

THAT THE D = S S^ IS MINIMUM IF 

M = 1 




Figure 4. Rule of Minimum Antecedents 
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Rules of Partial Ordering 

The rule of partial ordering is illustrated in Figure 
5. If two sets exist which have at least one common 
aggregating element (TB), then they may be combined. 
They should always be combined or attached in a 
manner such that the rule of minimum antecedents is 
supported. 



IF TWO SETS 

AND EXIST. 

AND IF THEY, BUT AT LEAST 
ONE COMMON AGGREGATION, I. E. 

0-^0 

THEY MAY BE COMBINED 




Estimating Program Size 

A frame is a unit of programmed sequence which 
contains a) the confirming response and a new stimu- 
lus, b) the confijming response, or c) the stimulus. A 
classical linear response frame would fall in (a), a 
branching or intrinsic program will consist of several 
(b)’s for each (c). 

In order to estimate the size of a program, one may 
write, for each programmed sequence, the number of 
frames expected for that sequence over the arrow (as 
shown in Figure 6). To improve the accuracy of pre- 
diction, one can also estimate the minimum and maxi- 
mum number of frames. 

If the program is not going to be homogeneously writ- 
ten, one may write the techniques which are planned 
under the programmed sequence. The hypothetical es- 
timate of a program’s size based on a behavioral net- 
work is shown in Figure 7. One accumulates to any 
point in the program for the entire program, the 
minimum expected, and maximum number of frames 



(?) — — >(T) 

^^TYPE OF SEQUENCE 

N = MINIMUM NUMBER OF FRAMES 
M = EXPECTED NUMBER OF FRAMES 

P = MAXMIMUM NUMBER OF FRAMES 
Figure 6. Description of Programmed Sequence 

expected. Generally speaking, one should have a min - 
imum of one TB for each 25 frames, or approximately 
a minimum of four TB’s for each hour of nmning time. 

Example of Application 

During the preparation of PERTeach training mate- 
rials a behavioral planning network was developed. 
This network was developed after a complete list of 
terminal behaviors was prepared. The list of termi- 
nal behaviors, the gathering of backup data, general 
pedagogical outlines, and decisions are usually made 
before frame writing has commenced. Figure 8 shows 
an excerpt of the behavioral planning network for the 
beginning of PERTeach. It is important to notice how 
some terminal behaviors which are taught late in the 
program could be taught much earlier if so required. 

It is evident in the analysis of any subject matter for 
a program that the order of the terminal behaviors is 
never linear. That is, seldom is any given terminal 
behavior followed by one and only one terminal be- 
havior. In fact, given a fundamental sequence one 
has many diverse paths to follow. Those paths chosen 
vary by the pedagogical and technical approach of the 




Figure 7. Hypothetical Estimate Based on a 
Behavioral Planning Network 
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programmer. The educational and pedagogical strength 
of the programmer are the only assurance that the al- 
ternatives are combined in an optimum manner. 

SUMMARY 

The usefulness and success with this technique leads 
us to describe it in more formal terms to allow for its 



further use and application. The use of a behavioral 
planning network ensures that an “erroneous set” does 
not mislead the programmer. This network is also im- 
portant in assigning and planning programming tasks. 
Knowing which sequences are dependent upon which, 
one can assign a programmer to write the first or initial 
sequence. At the point where there is a branch, one 




1.1 Student derives the acronym PERT 

3.1 Describes differences between PERT 
and Gantt-Milestone charts 

4.1 Defines PERT activities and events 

5.1 Student determines that an event can 
occur according to PERT rules 

6.1 Student determines that an activity 
can occur according to PERT rules 

7.1 When constructing a network, the 
student includes all activities and 
events 

8.1 When constructing a network, the 
student eliminates the non-essential 
events and activities 

9.1 Remove closed loops 

10.1 Defines a, m, and b 

35.1 Recalls that networking is a group 
activity 



35.2 Recalls need for clear identification 
of activities and events 

35.3 Recalls that group members should 
have common knowledge of the 
program objectives 

36. 1 Recalls basic right to left method 

37.1 Recalls basic left to right method 

38.1 Recalls and uses accepted symbols 

43.3 Recalls that updating occurs on 
two week schedule 

53.1 Managers use Security Check List 
and recall the importance of the 
PERT data to the enemy 

53.2 Managers recall that a PERT net- 
work may indicate information 
classified higher than the con- 
tractors portion of the contract 

53.3 Program ofiSce personnel recall the 
doctrine of reconstruction and how 
to apply it 



Figure 8. Portion of PERTeach Behavioral Planning Network 
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may assign a second programmer who now can learn 
the subject matter in question by taking the frames 
written by the first programmer. This dlows one to 
quickly add programmers and integrate tlieir styles as 
a natural function of how the program is constructed 
and not by fractionating and requiring that each one 
learn all the subject matter. The ability to estimate 
properly is necessary for the planning, scheduling, and 
costing of programming activities. It is important to 



try to break down estimates to natural units instead of 
trying to estimate a whole job by rule of thumb. The 
former type of estimating makes estimates more ac- 
curate. 

Behavioral planning networks hold promise as a tech- 
nique of viewing terminal behaviors before structuring 
a program. They require that a better learning theory 
be developed and more practice in their application 
occur before one can clearly state their usefulness. 
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IV. The Teacher 

EDUCATIONAL TECHNOLOGY AND THE ROLE OF THE TEACHER 

Joseph A. Tucker, Jr. 

With a reputation of being right in the post, Joseph A. Tucker, Jr., mokes some 
thought provoking predictions for the future, a refreshing redirection of our attention 
to the human element of the technology of education, the teacher and the adminis- 
trator. He summarizes: “. . . our education in the future will be greatly affected by 
new technological developments. However, these will be geared to support the 
teacher and the administrator as well as the student. The type of personal teacher- 
pupil relationship that we now have will be increased rather tlion be eliminated. 
Education will be individualized rather than standardized.” 



Dr. John H. Fischer, President of Teachers College, 
Columbia University in a few short paragraphs has 
listed the research and development areas of immedi- 
ate concern to educators and educational technologists. 
In his inaugural address in Riverside Church, New 
York City, this past fall. Dr. Fischer described Teach- 
ers College’s opportunities for work in advancing edu- 
cation as a professional human endeavor. He spoke of 
three broad fields. (I quote from Teachers College 
Topics) : 

First was the “opportunity to focus the energies of educa- 
tion on helping people to find meaning in the world and to 
fimd it in their lives”. The school, and the humanities in 
particular, can “expand horizons, deepen insights, sharpen 
sensitivity”. A second opportunity is the chance to investigate 
further &e processes of education. “The connection between 
the total pattern of the curriculum and what we now call the 
process of education offers an imjiortant and largely unde- 
veloped field for study. The things, the devices, the materiel 
that can make learning more readily and more widely available 
and teaching more efiicient comprise a third field of opportunity 
newly open to creative work”. President Fischer declared. 
The most urgent problem is to develop a theoretical approach 
to the design of educational systems to join the talents and 
skills of people operating at different levels of competence and 
responsibility with the capabilities of a variety of mechanical 
and electronic devices. “Giant strides are imperative here”, 
he stressed. 

The latter two opportunities are the subject of this 
paper. 

Both the educator and the developers of educational 
support equipment are guessing at the future. The 
incredibly rapid technolo.gical developments of the last 
decade, especially in computer technology and auto- 
mation have initiated much soul-searching about the fu- 
ture of our educational establishment. Radical change 
in educational processes and administrative practice 
has been predicted. 

Distinguished advocates of computer technology have 
envisioned a school environment in which the student 
working on an individual basis will have much of his 
learning process actually controlled by computer. It 



has been intimated that this can be accomplished by 
1970. This type of prediction along with the enthusi- 
astic but over-optimistic claims of the advocates of 
auto-instructional training have caused much concern 
in the educational co mmuni ty and a major problem in 
the pl annin g of new educational facilities. 

The central issue is the future of the teacher. Ilie 
concern of teachers about the impacts of new techno- 
logical developments, particularly automation, is jus- 
tifiable. If the student is to be controlled in the act of 
learning through inter-action with computers prima- 
rily, the teacher will of necessity have to act as a moni- 
tor of the learning process rather than a director of it. 
Computer control and large scale use of auto-instruc- 
tional materials require detailed long-range preplan- 
ning of instruction. It is this preplanning that lies be- 
hind projections of radical change in educational 
practice. This is not only a problem for the teacher. If 
this radical change occurs, it will greatly affect the 
form of products, including textbooks, that are pro- 
vided in support of education. 

I believe the prediction of radical change in the 
teacher’s role is in error. It results from looking at 
only a small portion of the great technological advances 
of the last decade and not at all of them. It is ironic 
that a future of control of learning by long-range pre- 
planning should seem to have logic and inevitability 
on its side, at a time when technology can make it 
possible for the short range, personal, clearly indivi- 
dualistic control of learning processes by the teacher to 
reach the great level of effectiveness which it has al- 
ways promised but has been unable to obtain due to 
impossible teacher workloads. Technology could pro- 
duce “radical sameness” rather than radical change in 
educational practice. 

A major characteristic of present educational prac- 
tice is that individuation of instruction is a preroga- 
tive of the teacher. Granted that there is and must be 
much pl anning of educational content for various grade 
levels and that textbooks, workbooks and tests are 
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based on this planning, one must recognize that the 
actual conduct of the class is based on the teacher’s 
pl annin g of the work to be accomplished and her exe- 
cution of that planning. Though this is far from a per- 
fect system, it is clearly differentiated from one in 
which the teacher is essentially a monitor rather than 
a director of the educational process. The behavior 
scientists who speak of auto-mstructional training pro- 
cedures and computer control of the learning process 
have concentrated on the activity of the student. Few 
of them have as yet given thoughtful consideration to 
the role of the teacher. Our rapidly expanding knowl- 
edge of how to integrate programmed lessons with 
other instructional media is largely the result of work 
by educators and teachers. The educational technol- 
ogist must support this work. Major emphasis should 
be given to improving educational procedure through 
improving the conditions by which the teacher can 
control it. TEACHERS ALONE CAN PROVIDE 
THE TYPE OF INDIVIDUALIZED INSTRUCTION 
THAT AUTO-INSTRUCTIONAL AND COMPUTER 
BASED LESSONS SEEM TO PROMISE BUT CAN- 
NOT POSSIBLY ACCOMPLISH. It is necessary to 
find ways through which the teacher can do the same 
things that she is doing now far better so that she can 
truly attend to the individual differences that exist 
among her students in a learning sense as well as in a 
guidance sense. Educational technology must in- 
clude in its mission the increasing of the efficiency of 
the teacher in a manner analagous to medical science 
increasing the efficiency of the doctor * at of itself not 
destroying the individual relationship that can exist 
between a doctor and his patient. There is a need for 
techniques and equipments which will help the teacher 
individualize instruction for each student. 

Both law and educational tradition have fixed the 
available time that a student has to get an education. 
Educators have worked constantly to maximize the 
effective use of available training time. Much of this 
planning, in the curricular sense, has been concerned 
with determining objectives of education. As a result, 
we now have many curricula for our students. We 
have greatly improved our facilities for all educational 
purposes. The intent is to provide an environment in 
which learning can be accomplished more efficiently 
than in past environments. In spite of criticism, the 
objectives as defined by educators have been met ac- 
ceptably. We do make good use of available training 
time, but we can make far better use without radical 
change in our educational methods. I believe that the 
next great gain in the efficient use of available train- 
ing time will occur in the improvement of the admin- 
istrative practices within the classroom. The poten- 
tiality of educational technology to save learning time 



by improving the teachers administration of learning 
has been lost in the excitement over teaching machines, 
programmed texts and other innovations aimed at im- 
proving the act of learning itself. Though these new 
techniques are important they cannot be effectively 
introduced until the teacher can perform more effi- 
ciently. 

Before discussing how technology can effect and im- 
prove teacher-pupil communicative relationship and 
administrative practice in the classroom, I will dram- 
atize the significance of improving the use of training 
time at the classroom level by the following oversim- 
plified but logically correct projection. If classroom 
efficiency can be improved so that five minutes of ad- 
ditional learning on an average can be achieved by 
the students of each 50-minute class, these students 
will accrue three weeks additional training time in 
each academic year. Continuing this projection through 
10 years of education from the first grade on, one can 
note that the total curriculum can be accomplished 
one year sooner than at present, making one year of 
additional training available to each student. At a 
period when great pressures exist to lengthen the school 
year to provide our students with more education, 
the possibility of accomplishing it within present time 
allotments by technological support of the teacher re- 
quire significant study. 

Technological support of the teacher can provide sig- 
nificant improvement in current educational practice 
in two areas. One is in her control of group processes 
in the classroom; the other, a corollary of the first, is 
in the planning of individual assignments for her stu- 
dents. 

Group training can be improved by removing many 
of the inefficiencies in our present methods. These in- 
efficiencies are really simple procedures to which we 
have become so accustomed that we do not realize 
the amount of time they waste during any school day. 
Much of the time of a typical period is spent in “proc- 
essing” activities, in moving paper about, in tabulating 
results, in “housecleaning”. Technology can make 
these activities almost instantaneous. The quizzing of 
students by the teacher, even though they are in a 
group, can be made private as well as public. Re- 
sponding of the students, both as individuals and as a 
group, to questions from the teacher can be accom- 
plished easily and quickly without the time consuming 
counting procedures we now use. Inquiries of the 
teacher by a student can be made a private matter 
rather than one that disturbs the whole class, imless 
it is in a group discussion or class participating event. 
Individual assignments can be given to students quickly 
and efficiently without using class time to explain to 
each student what it is he is to do. Tests can be scored 
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automatically so that the teacher is not involved in 
their processing. All of this is possible today with the 
proper blending of available equipment and can be 
accomplished far more inexpensively than educators 
have been led to believe. 

The idea of individualizing instruction to meet the 
needs of each student does not mean that each must 
have his own unique curriculum. All systematic meth- 
ods of education, including the auto-instructional tech- 
niques, which are really group training methods 
administered individually, assume that there is a curric- 
ulum with subject matters which are to be taught 
according to a prescribed sequence. This is as it should 
be. Consequently, the concept of adjusting to indivi- 
dual differences means, frequently, the making of 
minor alterations in the prescribed sequence in order 
to meet the immediate needs of the learner. 

If one were to ask a parent of a high school student 
whether or not his child should receive ten minutes 
of individual attention each week from his teacher, say 
his history teacher, the parent is most likely to answer 
“yes” without thinking very much about the signifi- 
cance of a “yes” answer. In fact, one might almost as 
quickly answer “yes” if the question were stated “Do 
you believe every child learning history from his 
teacher should receive ten minutes individual atten- 
tion per week?” In answering “yes” to a question like 
this, we do not stop to realize that what we are saying 
is that a typical high school teacher should spend 25 
horns per week at a minim um providing individual in- 
struction to her students. With that fact in mind, let’s 
consider the question again. Does your child deserve 
ten minutes of individual instruction from his teacher 
each week? We should think hard before we answer 
this time. To answer “yes” is to be impractical since 
teacher time is not available. How about ten minutes 
every two weeks? This might be possible if the teacher 
is properly supported. The next problem then is to 
make maximum use of ten-minute each two week diag- 
nostic session. 

In order for the teacher to react to the individual 
progress of a student, she must have information as 
to his strength and weakness on a timely basis. The 
first prerequisite, for individualized training, is effi- 
cient assessment of student progress. This must be 
accomplished in a manner that requires very little 
time. The second requirement is the actual prepara- 
tion of an individual assignment for each student by 
the teacher. This must be accomplished quickly and 
efficiently since a typical high school teacher will have 
at least 150 pupils that she contacts each day. Conse- 
quently, one of the marks of her professionalization 
will be that she has such a grasp of her subject mat- 
ter and the materials available for her students, that 
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she can quickly evaluate the students* performance 
from the records provided her and make recommenda- 
tions for the next two-week period. She will have to 
prescribe quickly and efficiently much as a doctor 
prescribes. To do this she must have methods of pre- 
paring assignments which do not require her to spend 
considerable time writing out instructions. Further, 
she must not be required to locate the materials for the 
student herself. The third requirement is for an effi- 
cient way of transmitting the assignments of the stu- 
dent. This again must be accomplished quickly and 
efficiently with a minim um of time expenditure on the 
part of both teacher and pupil. Lastly, the pupil must 
be able to get materials required to accomplish his 
special assignment as well as his regular assignment 
with minimum time involvement on his part. AH of 
the essential technological parts to make this type of 
education feasible now exist. They must be integrated 
into an efficient component of our educational enter- 
prise. 

The achievement of complete capability for sup- 
porting education in this manner will take about 10 
years. The initial systems will be crude compared to 
those that will come later. However, they will have all 
of the essential parts required to support the teacher. 
The requirement is for equipment which will assure 
efficient use of teacher time. The purpose is to maxi- 
mize the teacher’s control of learning processes both 
on a group and individual basis. The need is for a 
flexibility in educational environment, materials and 
media that can match the flexibility of individual 
teachers. 

I am certain that during the next five years, educa- 
tors will begin to use microtechnologj'', including eight 
millimeter film and accompanying devices to provide 
educational materials to students. For example, there is 
a photographic technique that permits the storing of up 
to 10,000 images of information on a one 6 by 9 inch 
card. Such a card can lead to a complete revolution 
in the method by which educational materials are dis- 
tributed and handled by the school. This card and its 
scanning mechanisms can be made compatible with 
equipment which can reproduce the materials scanned 
in as many copies as may be required for a specific 
use. It should be repeated that all of the equipments 
needed to do this exist now. This will mean that mate- 
rials for students will not have to be given to them in 
lump in the beginning of the year, but can be distributed 
and reproduced for them as needed. Such a capa- 
bility permits great flexibility in the preparation of 
assignments for individual students and leads to true 
individualization of instruction under the tutelage of 
the classroom teacher. In any case, it is apparent that 
combinations of new educational equipments will lead 
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to a further professionalization of the teacher’s role by 
bringing the teacher in more intimate contact with the 
learning activity of each student. Yet these new equip- 
ments in no way interfere with the conduct of instruc- 
tion in traditional manners when this is desired, as it 
may frequently be. They allow for individual differ- 
ences among teachers as well as among students. This 
is desirable. Each teacher can decide which of the 
new media best fit her own style of teaching. A stand- 
ardized teacher is no teacher at all. And what is a 
standardized student? Standards must remain but 
standardization must go. 

Computers of varying capacity will have an impor- 
tant role to play in the application of new technology 
to improve classroom management. However, it is un- 
likely that computers will be used to control students’ 
learning as has been suggested in the past few years. 
Computer capacity can never be used effectively for 
this purpose. It is impossible practically to accomplish 
the t 5 q)e of programming that would be required to use 
the computers’ capacity to respond to individual dif- 
ferences since the programming task would be never 
ending. Much simpler devices can be used to present 
the auto-instructional programs that can be developed. 
Consequently, the control of learning behavior by com- 
puters so far as conceptual learning materials is con- 
cerned is highly unlikely. However, the computer will 
be very important in making possible a significant in- 
crease in the instructor’s ability to individuate learn- 
ing. The computer can handle the data processing 
parts of this task better than any other available 
means. Computers can control the preparation of stu- 
dents’ records, grades, and the like. Ungraded schools 
for a large number of students can only be accom- 
plished with the help of computer technology which can 
keep track, on a day-to-day basis, of the progress being 
made by the student. In this way, the computer is a 
helper to the administrator and to the teacher. It 



makes possible the accomplishment of new techmques 
of education which educators understand thoroughly 
but have not previously been able to implement prac- 
tically. 

Computers will be used directly to control the learn- 
ing of certain types of motor skills. In this, computers 
are not replacing the function of a human tutor. They 
are performing a function that no human tutor can per- 
form. This “motor skills” computer-based training 
technology is embryonic and probably will take 20 
years to develop since it requires use of computers of 
a type about which we have very little knowledge at 
this time. 

I am sure that you all recognize that I am talking 
about a “systems” approach to the development of an 
effective learning environment. The theme is to max- 
imize the use of available trainng time. This can be 
accomplished by maximizing the effectiveness with 
which all available resources are used. The most im- 
portant resource of all is the teacher. Yet the teacher 
is the resource that we use least effectively. Educa- 
tional technology must produce methods and devices 
that maximize the effective use of our resources. It 
must provide instructional aids in support of the ad- 
ministrator, teacher, and student. For much of the 
curriculum, the need is not for equipment for which one 
can claim technological wonders in increasing the 
efficiency of the learning process, but rather for those 
which, while permitting use of the latest educational 
techniques, wiU improve use of available training time 
by eliminating clerical tasks and poor classroom logis- 
tics, providing better use of available training space, 
permitting more individualized teacher performance, 
simplifying record keeping, and permitting more flexible 
curriculum planning. Gains in learning wiU result in- 
directly due to improvement in the teacher’s work situa- 
tion and directly from the effect of increased individual- 
ization of instruction. 
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THE IMPACT OF PROGRAMMED INSTRUCTION ON CONVENTIONAL INSTRUCTION 

W. James Popham 

W. James Popham poinfs ouf fhaf programmed insfrucfion will take several years 
before it will make serious inroads on the usual instructional methods of public schools. 

But when it does, it will achieve greater Instructional proficiency because of its sig- 
nificant and salutary impact in six areas: explicit educational objectives, appropriate 
student practice, student knowledge of results, concern for individual student differ- 
ences, responsibility for learning and curriculum content. W. James Popham discusses 
the impact of programmed instruction in each of these areas. 



A person experienced in programmed instruction 
undoubtedly would be in a position to make several 
fairly safe predictions regarding the future of this 
area. One highly defensible prophecy would be tliat 
programmed instruction methods will never totally re- 
place more conventional instructional procedures. In 
fact, it will take a few years until programmed ap- 
proaches make any serious inroads in the usual instruc- 
tional methods of public and private schools. As a 
parent of school age children, and a programmed in- 
struction proponent who has been somewhat less than 
ecstatic over the quality of currently available pro- 
grams, I confess I do not view this prophecy with total 
dismay. 

Actually, this kind of prediction could have been 
made some time ago with equal assurance of fulfQl- 
ment. It occurred to me several years earlier as I was 
caught up in a swirl of enthusiastic excitement about 
the potential of programmed instructional procedures, 
that conventional classroom instructors could profit 
tremendously from studying the methods of the pro- 
grammer. And since it was patently clear that the 
classroom teacher would have to v/ait years until a 
sufficient number of suitable programs were available 
for his use, it seemed a tragedy that certain techniques 
of programmed instruction could not be infused into 
the pedagogical repertoire of the conventional teacher. 
I, therefore, have found myself increa^rngly drawn to 
this task. 

Now someone could correctly observe that most of 
the elements of programmed instruction have been 
known to psychologists and educators for many years 
and that it’s about time teachers started making use of 
them. It seems to me that in order to change the be- 
havior of teachers, and for that matter the teacher of 
teachers, something more dramatic than esoteric, al- 
beit conclusive, research reports was needed. Pro- 
grammed instruction has supplied that drama. 

It is not too uncommon for an instructor who teaches 
an introductory course in programmed instruction to 
have teachers in his class, at the conclusion of the 
course, remark, “I may never be a programmer, but 
this course will make me a better teacher.” Why should 
this be so? In other courses in a teacher education 



curriculum, courses which are ostensibly designed to 
improve the teacher’s instructional proficiency, such 
comments are rarely forthcoming. I believe that a 
teacher’s exposure to the methods of programmed in- 
struction can provide a stimulus of sufficient strength 
to affect a teacher’s usual instructional methods. More 
important, programmed instruction is shaping the be- 
havior of instructor trainers and teacher educators. 
Such influential individuals are now beginning to in- 
corporate programmed instruction principles in their 
own teaching repertoires and to advocate their students’ 
use of them. 

It is true that such influence is only beginning to be 
felt, but in a relatively short time I believe we shall 
receive significant classroom dividends wherein the 
teacher faced with 30 grade school children, 35 high 
school pupils, or 100 college students, will manifest 
greater mstructional proficiency because of the impact 
of programmed instruction. 

It seems to me that there are at least six easily dis- 
cemable instances of influence that programmed in- 
struction has begun to have on conventional mstruc- 
tion, influences that will become markedly stronger 
in the next three or four years. I shall briefly discuss 
each of the mstances, recognizing th.it other attributes 
of programmed instruction may in time make very 
striking contributions to conventional teaching proce- 
dures, perhaps even more significant than those which I 
shall identify. 

The first, and to me the most significant, impact that 
programmed instruction has had on conventional in- 
struction is in the field of instructional objectives. As 
we know, programmers generally specify instructional 
objectives in terms of student behavior. Classroom 
teachers generally do not. Instead, classroom teach- 
ers usually preface a description of their instructional 
plans with high sounding but wholly ambiguous asser- 
tions such as “The student should improve his ability 
to meaningfully deal with the underlying causes of the 
Civil War.” 

Of course, such loosely specified educational aims 
offer little assistance to the instructor, either for the 
selection ci learning activities or for the evaluation of 
mstructional effectiveness. 
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Now even though Bloom’s carefully detailed Taxon- 
omy of Educational Objectives has been available to 
educators for seven years, few classroom teachers have 
indicated an awareness of the need for operational in- 
structional goals. 

Currently, with the focus of programmed instruction 
on explicit objectives framed in terms of student rather 
than instructor behavior, I note an increasing recog- 
nition on the part of public school personnel that there 
are strategic advantages in stating specific instruc- 
tional objectives. More important, educators are be- 
ginning to stress the value of such objectives in their 
pre-service and in-service professional classes. 

No small part of the credit for this development 
should go to programmed instruction specialists such 
as Robert Mager, who have emphasized the need for 
behavioral objectives in their writing and public speak- 
ing engagements. Mager’s excellent little volume on 
how to prepare instructional objectives has been used 
extensively during recent months in a number of teach- 
er education programs. I stress the criteria for well 
stated objectives in a required curriculum and instruc- 
tion course. One must assume that some of this at- 
tention to objectives will eventually be translated 
into a teacher’s behavior. It is important to note that 
just a few year.- ago, that is, before the impact of pro- 
grammed instruction, this emphasis of explicit objec- 
tives was not present at many teacher education insti- 
tutions. 

A second influence on conventional instruction, which 
is in large measure attributable to programmed in- 
struction, involves the necessity of having the learner 
make active responses, which are appropriate to the 
instructional objective in question. Far too often con- 
ventional instructors, who really have most desirable 
instructional goals in mind, design learning activities 
for their students which are, at best, only peripherally 
relevant to these goals. The programmer soon discovers 
that if he wants a learner to engage in terminal behav- 
ior X he must provide many opportunities throughout 
the program for the learner to practice behavior X. 
We cannot build a program featuring many instances 
of the students’ use of behavior Q and expect him to 
leave the program imbued with behavior X. As F. 
Rand Morton has cogently observed, through the use of 
an exclusively verbal or textual program you may be 
able to teach a prospective sales clerk that she should 
smile at customers, you may even make her want to 
smile at customers, but to make her smile at custom- 
ers you must build a program that allows her to receive 
reinforcement for actually smiling. In other words, the 
program must contain opportunities for the student to 
engage in appropriate practice. 



Thirteen years ago Ralph Tyler in his University of 
Chicago curriculum and instruction course was urging 
precisely the same thing. Why has it taken program- 
med instruction to change the behavior of the conven- 
tional instructor? Perhaps for the same reasons we 
are told a programmer must condition a few rats or 
pigeons before he can comprehend the true power of 
behavior-shaping principles, so, too, a conventional 
instructor may have to try his hand at changing an 
individual student’s behavior by preparing a system- 
atic program before he realizes the significance of stu- 
dent response, and for such responses to be relevant to 
the instructional objectives. You don’t have to fail too 
often at programming fairly simple instructional ma- 
terial before you start looking for explanations. Quite 
often the teacher discovers the real culprit in his un- 
successful program is the lack of appropriate practice. 
Such notions, fortunately, are beginning to show up 
in the conventional classroom. 

A third principle of programmed instruction, which is 
being incorporated into conventional instructional ap- 
proaches, is that of immediate knowledge of results. 
The notion that the student should learn quickly of the 
appropriateness of his responses is almost universally 
practiced in programmed approaches. It is not surpris- 
ing, therefore, with teachers being increasingly ex- 
posed to programmed instruction, tiiat the knowledge 
of results principle should find its way into conventional 
classrooms. This is perhaps the most easily incorporated 
programmed instruction principle, and part of this re- 
cent increase in teachers’ use of the principle is un- 
doubtedly a result of the fact that they can introduce 
it so easily to their regular classroom procedures. I 
note from a research study that student teachers of 
a certain university who have been urged to used this 
principle do employ it in their classrooms with greater 
frequency than student teachers for whom this princi- 
ple was not emphasized. Personally, I do not consider 
this principle one of the greatest contributions made by 
programmed instruction, but I do believe the increased 
influence with which it is used in regular classrooms is 
directly attributable to its widespread employment in 
programmed instructional methods. 

The fourth area of impact which programmed in- 
struction has made on conventional instruction is 
quite different from the former three. In this instance, 
the stress on self-pacing or individual differentiation 
seen in programmed instruction has not been trans- 
lated into conventional instruction practice. True, 
classroom teachers have urged individual differentia- 
tion for years, but few have really tried to implement 
this principle. 

Fortunately for a teacher’s tranquility of conscience, 
the gross injustices done to exceptional students in a 



non-differentiated conventional classroom rarely are 
clearly evident. With the self-pacing features of pro- 
grammed instruction, such atypicality often results in 
startling reductions in the time supposedly required 
to master a subject. Teachers cannot hear the fairly 
frequent accounts of how bright students finish whole 
year courses in half the time without being struck by 
the fact that allowances for such differentiation are 
rarely made in the conventional classroom. In other 
words, with respect to individualizing instruction, pro- 
grammed instruction has induced a great many guilt 
feelings among conventional instructors. So far, that’s 
about all it is, guilt, but, with time and conscience 
qualms mounting, I believe we can expect greater con- 
cessions to self-pacing in the conventional classroom. 

The fifth significant contribution that programmed 
instruction has made to conventional instruction re- 
lates to the implicit reliance on student testing to im- 
prove the quality of programmed instructional mate- 
rial. As we know, programmed instruction has largely 
shifted the responsibility for the quality of instruction 
from the pupil to the programmer. Many conventional 
teachers are impressed with the fact that programmers 
are willing to accept this burden of proof and put their 
instructional effectiveness on the line by testing a pro- 
gram’s quality in terms of student behavior. 

Certain classroom teachers are actually beginning to 
view unattained instructional objectives not as the 
fault of “inattentive students” but as a reflection of 
their own instructional inadequacies. Whatever influ- 
ence programmed instruction has in promoting this 
type of attitude will yield remarkable benefits to many 
pupils, pupils who will profit from a teacher who is 
willing to modify his instructional behavior if his stu- 
dents fail to reach the heights he had in mind for them. 

The sixth contribution of programmed instruction to 
conventional instruction, which I shall designate as 
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“curriculum concern”, may well have the most far- 
reaching effect upon the schools. I became aware of 
this sixth point when my students in introductory pro- 
grammed instruction courses would voice a w’de-eyed 
observation: “You know, this method works so well 
we’ve really got to be careful about what we program!” 

True enough, as any instructional nv?.thod becomes 
increasingly effective, a proportionate increase in at- 
tention must be given to its content and objectives. 
One of the consoling features of conventional instruc- 
tion is that it is often so ineffective that we don’t have 
to worry too much if the WTong curriculum is employed. 
As instructional methods, both programmed and con- 
ventional, become more capable of achieving their 
objectives, we shall see an even greater concern regard- 
ing the appropriateness of the curriculum. 

Already there have been several instances when pro- 
grammed instruction experiments have been criticized, 
not for the qualit}!^ of the programs used, but for the 
content selected. The same wiU be increasingly true 
with conventional approaches as they becoLie more 
effective. 

In summary, I have briefly examined six contribu- 
tions which programmed instruction has made to cur- 
rently employed conventional instruction techniques. 
Some educators fear that programmed instruction will 
have undesirable influences on conventional instruction, 
and it will! Lazy teachers will rely too heavily on poor 
programs. Economy-oriented administrators will try 
to spread over-worked teachers too far by using self- 
instruction materials. Yet, even though these unsound 
uses of programmed instruction will occur, and taking 
no account of the obvious instructional benefits of good 
programs themselves, I am thoroughly convinced that 
the residual impact of programmed instruction v/ill be 
significant and salutary. 



A VENTURE IN PROGRAMMED INSTRUCTION 



Sister M. Dorothy, C.D.P. 

Sister M. Dorothy reports on her trial of TEAAAC Basic Math, 3600 I frames, and 
TEMAC Algebra I, 8000 I frames in her eighth grade at St. Martin Hall Grade School, 
the campus laboratory school of Our Lady of the Lake College in San Antonio, Texas. 
She indicates its use was successful and points out both the advantages and pitfalls 
of the use of programmed instruction from the point of view of the students as well 
as the teacher. 



An experiment is now being conducted in eighth 
grade mathematics at St. Martin Hall Grade School, 
the campus laboratory school at Our Lady of the Lake 
College in San Antonio, Texas, in conjunction with the 
Education Department, on the comparative efficiency 
of programmed instruction with the traditional type 
of mathematics instruction. A control group of students 
was selected in another eighth grade in another school 
having a comparable number of students. Intelligence 
tests and mathematics achievement tests were given 
to both groups before any instruction began. Five weeks 
later, copies of TEMAC Basic Math, published by En- 
cyclopedia Britannica, were distributed to 37 eighth 
grade students in the experimental group. 

The use of this new type te;d was carefully explained 
to the students. Mathematics was scheduled for 50 
minutes daily; however, the use of the TEMAC books 
was not restricted to the daily class period. Students 
were told that they might take the books home although 
there was no obligation to do so. Tests were to be taken 
upon completion of a specified number of frames and 
when students felt ready for them. During the mathe- 
matics period the teacher was available for help, di- 
rection, explanation, or just plain encouragement. Very 
few children needed or asked for help. Only twice, near 
the beginning of the course, was there a general class 
discussion of the problems in the texts. The students 
were advised and soon realized that when they came 
upon a problem that was troublesome or that they did 
not understand, the wisest thing to do was to go back to 
the frame in which the material was first presented. 
After rereading the explanation and reworking the prob- 
lems if necessar>r, students were generally able to pro- 
ceed without trouble. In the cases where this was im- 
possible, individual help was given. 

The majority of the students proceeded confidently 
and steadily through the 3 volumes of Basic Mathe- 
matics covering some 3600 frames. The first two stu- 
dents finished Basic Mathematics in five weeks. After 
21 weeks, 35 students had completed the 3 volumes 
while two students had not completed the third and 
final volume, but had completed the first two. Upon 
the completion of the Basic Mathematics the students 
were given a list of approximately 75 pages from the 



traditional mathematics text ordinarily used in the 
eighth grade. These pages included some of the mate- 
rial covered in TEMAC, but since it was presented in 
a different way it served as a review. The types of 
problems included insurance, taxation, mortgages, and 
various other problems involving the application of per- 
centage, geometric constructions, and measurements 
and a section on introductory algebra. 

After the material was satisfactorily completed, 
students were allowed to begin TEMAC Algebra I. 
This course consists of 5 volumes having in excess of 
8000 frames and 30 unit tests. About half of volume 5 
is devoted to a cumulative review of the course. There 
are 6 more tests to accompany this section. 

By the end of the school year, 2 students had com- 
pleted the course in Algebra I, while 20 others were at 
various stages of completion. Of this number, 3 stu- 
dents were working in volume 4; five were in volume 
3; and the other 12 students were evenly divided be- 
tween volumes 1 and 2. Time did not permit these stu- 
dents to complete the program. Motivation for these 
eighth grade students did not seem to present any dif- 
ficulty. When the books were distributed, the students 
were told that there would be no definite daily assign- 
ment and that they were to be on their own. Some set- 
tled back in their desks while smiles reflected complete 
approval of texts and procedures. Diligence soon re- 
placed any tendency to lazy complacency and with 
few, if any, exceptions the children made consistent 
efforts to use the programmed texts efficiently. 

The very nature of the texts afforded motivation by 
its immediate reinforcement of each fact or concept 
learned. The interweaving of instruction, drill, and 
testing held the children’s interest and there was no 
appreciable tendency toward boredom. In order to 
help forestall such an eventuality, students were told 
that when they came to a group of problems drilling on 
a process they knew, to work the first 3 or 4 problems 
and, if correct, to skip the rest. 

Although an attempt was made to eliminate human 
competitiveness as a motivation factor, by telling the 
students not to worry if a classmate was ahead, no one 
wanted to be left far behind. Another important mo- 
tivating factor was the feeling of accomplishment upon 



the completion of a book or volume. A certain feeling 
of justifiable pride was felt by the individual as he put 
aside one volume and began another. 

Chart I gives a graphic view of the number of stu- 
dents finishing Basic Mathematics in a particular num- 
ber of weeks. In examining this chart, it may be inter- 
esting to note that the 2 students who finis hed in 5 
weeks were outstanding students. The next group, who 
finished during the 7 to 9 week period were very good 
to average. Not aU of these 8 students attained the 
highest scores on the tests nor on the achievement tests 
given in September and in J;inuaiy. At least 2 of those 
in the 20 to 21 week group had an excellent mathemat- 
ics record in previous grades and they also had high 
scores on the achievement tests. This seems to point 
up the ever present problem of speed vs. accuracy. One 
student clearly recognized this problem. After several 
very low test scores, he saw his error in working too fast 
and asked to repeat almost an entire volume. This ex- 
perience helped not only the individual concerned, but 
also the entire class. 

A further study of this chart leads to a consideration 
of several problems involved in the use of programmed 
texts. Students completed the program at various 
thi^es; individual help was needed from time to time; 
and tests were administered at different times with dis- 
cussion of errors a must. Students who completed pro- 
grams had to be given interesting and challenging mate- 
rials to follow up the program and to provide sequential 
learning. Adequately meeting these needs required full- 
time attention to detail. 

Another problem was grading and reporting to par- 
ents. Since the students were working on an individual 
basis, it was impossible to give them a percentage or 
letter grade in relation to other members of the class. 
In order to give parents a picture of their child’s prog- 
ress, a mimeographed report was inserted in a regular 
report card, showing the possible number of points per 
test, the number of points made by the child, and the 
equivalent percentage grade. 



Number of Students 


Percentage Average 


A* 


4 


95.5 - 97.5 


B* 


8 


90.2-94.5 


C* 


15 


85.4-89.8 


D* 


8 


81.0-84.5 




1 


79.8 




1 


66.0 



Percentage averages on 8 tests in TEMAC Basic 
Mathematics 

* For comparison in using Table II 
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Table I shows the average percentage grades made 
on the 8 tests which accompany Basic Mathematics. 
The numbers of students falling within particular per- 
centage intervals have been indicated. These figures 
give a nearly perfect curve with the largest number fall- 
ing within the 85.4 to 89.8 interval. 

Table II shows the progress of those students who 
are now working in algebra. The letters A, B, C and D 
indicate the average intervals taken from Table I for 
each student in Table II. With two exceptions the stu- 
dents made satisfactory progress. The student from 
group “A” with an 83.0 in algebra may need to repeat 
or reread part or all of volume 1 in algebra. The girl 
from group “D” who was extremely eager to begin 
algebra worked slowly, but with determination, and 
repetition seemed necessary at each step. Her under- 
standing and progress had to be carefully checked. 

As students finished Basic Mathematics and took up 
the traditional text, a carryover from the programmed 
text was apparent. For the most part they seemed 
capable and desirous of carrying on as they had done 
in TEMAC. Explanations and problems were carefully 
read and attempted. A gratifying feeling of independ- 
ence and self-confidence seemed to have developed and 



Tests Taken 


Average on 
Algebra 
Tests 


Average in Basic 
Mathematics 


30 


96.4 


A 


30 


95.8 


A 


26 


96.2 


B 


22 


92.5 


A 


19 


94.7 


B 


16 


91.2 


A 


16 


92.4 


C 


10 


92.6 


B 


10 


91.0 


B 


9 


89.3 


C 


O 

u 


88.8 


B 


8 


83.6 


B 


7 


85.4 


C 


7 


94.5 


B 


7 


90.0 


C 


6 


73.0 


D 


5 


89.2 


C 


4 


94.0 


C 


4 


89.7 


C 


4 


88.8 


C 


3 


83.0 


B 


3 


99.3 


C 


Percentage averages on TEMAC Algebra tests 



Table I Table II 
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they approached word problems and others without the 
usual hesitancy. 

An examination of Table III shows the grade place- 
ment gain of the students from September through 
May. The interpretation of these data may be mis- 
leading imless it is borne in mind that the lowest 
amount of growth is shown for students who scored 
very high on the tests in September. Since the tests. 
Sequential Tests of Educational Progress, are designed 
specifically for the use of eighth grade, it is possible 
they do not make adequate provision for the high 
achievers. On the other hand, the greatest gain is 
shown by students who made the lowest scores on the 
first test. Many times this question is asked, “How 
do others at St. Martin Hall, teachers, parents and 
students, feel about programmed instruction?”. The 
other teachers are very much interested in the results of 
the eighth grade study and they are eager to try out pro- 
grams on their own level. If and when programmed 
instruction becomes the accepted and usual practice, 
propaganda or persuasion techniques will not be neces- 
sary for this faculty. 

Parents’ reactions have varied from enthusiastic to 
skeptical. Most of them seem to have adopted a “show 
me” or “FU wait and see!” attitude. Following a very 
interesting and enlightening discussion of programmed 
instruction during a P.T.C. meeting, one parent seemed 
to reflect the feeling of the group in these words, “Fm 
sure this type of teaching must be good but I don’t 
really understand it. When my 3 daughters were in 
8th grade I knew exactly what they were doing, but I 
have no idea of how or what my son is doing”. 

What about those most directly concerned, the eighth 
graders themselves? Their reactions also are varied 
and I would like to share some of their comments with 
you. In attempting to find out how the eighth graders 



Grade Placement Gcun 


Number of Students 


.0- .4 


4 


.5- .9 


10 


1.0- 1.4 


11 


1.5 -1.9 


7 


2.0 -2.4 


3 


2.5 - 2.9 


2 


Grade placement gain from September to May 



Table 111 



felt about programmed texts, several lead questions 
were proposed. When asked if they liked programmed 
instruction, all 37 answered in the afltonative. Four 
students answered “no” to the next question, “Would 
you like programmed texts in other subjects?”. The 
other 33 students expressed a desire for programs in 
every subject and especially in English and Science. 
The last question asked for a brief statement setting 
forth reasons for liking or disliking programmed texts. 
Some of their comments were: 

Sophie: “I like TEMAC because it taught me new ways of 
adding, subtracting, multiplying, and dividing. The one thing 
I did not like, was not having a definite assignment every 
night. If I don’t have an assignment, I don’t work”. 

Tally: I think I learned a lot. You can go at your own 

pace. You do not have to wait for others or if you want to 
go slow no one pushes you through”. 

Susan: “I like the mathematics and algebra. When I have 
worked for a while I feel like I have accomplished something”. 

Richard: I like this course in mathematics for many 

reasons. Mainly because we do not have regular classes; we 
do not have to be teacher-taught unless we have trouble; 
homework is not required; and we may go at our own speed. 

I think our eighth grade has an advantage over many schools”. 




Becky: “I like TEMAC because you may go at your own 
speed without hindrance and you may take your tests whenever 
you want them”. 

Priscilla: “I think you can learn more with TEMAC if you 
are willing to work. It enables you to work quickly through 
the easier parts and slowly through the more difficult parts”. 

Joe: “The biggest help in using TEMAC is that the minute 
you make a mistake you are corrected”. 

Judy: “I used to be very poor in mathematics, but with 
these books I have really brought my grades up”. 

Cynthia: “I think programmed subjects are fine except for 
one thing: they leave out class discussion, an important factor 
in learning. One subject only, such as mathematics, should 
be programmed”. 

Michele: “I think that programmed instruction was an 
interesting way to learn mathematics. Although sometimes 
you get very tired of going over the same problem more than 
once, I suppose it’s necessary since you don’t have a teacher 
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giving out extra practice problems. I feel that I have learned 
a great deal this past year. I’m sure it would be very helpful 
to have programmed instruction in other courses, too". 

Mary Ellen: “I enjoy programmed instruction. The way 
the material is presented helps me to grasp it much more 
easUy than an ordinary text, which I need a teacher to ex- 
plain. There is the additional fact that it is possible to go 
over your own grade level to more advanced work”. 

In conclusion, I should like to say that I am happy 
to have had the opportunity of using programmed in- 
struction. While I realize very well that it is not an 
educational panacea, I am fully convinced that it has 
many possibilities and tremendous potentialities. As 
educators, whether in the position of teacher or of 
administrator, I do not think that we can afford to be 
ignorant of the aspects of programmed instruction. 



71 






PROGRAMMING A COURSE IN WORLD HISTORY 

Robert La Fontaine 

Robert La Fontaine, a public school teacher, describes some of the problems in 
social studies areas, a tentative approach built on programmed instruction, and the 
effectiveness of an extensive history program he is writing and using in his own classes. 



The field of social studies and in particular the area 
of history has been, for the most part, generally ignored 
by those engaged in the creation of programmed mate- 
rial. Some have claimed that the social studies are too 
flexible to be properly programmed. And yet, if any 
subject area presented in today’s schools needs a new 
me^um of instruction, it is most certainly the social 
studies. In no other area does one find such large text- 
books and so little real learning taking place as in the 
area of history and related subjects. 

The blame for this unfortimate situation cannot be 
placed upon the textbook publishers. They are produc- 
ing textbooks to fit the requirements of the schools. 
Unfortunately these requirements were formulated back 
in 1914, almost fifty years ago. As a result, teachers 
today are faced with a tremendous volume of material 
to teach, with too little time, in most cases, to teach it. 
The state of Texas is a good example. In the seventh 
grade, students spend a full school year studying two 
hundred years of Texas history with time to spare. In 
the eighth grade the year is spent studying four hundred 
years of United States history. Ninth or tenth graders 
spend the same amount of time studying five thousand 
years of world history. As you can see, between Amer- 
ican and world history there is over a one thousand 
per cent increase in time span. This pattern of social 
studies courses is repeated in almost every state of the 
nation. Particularly in the field of world history with 
its tremendous volume of material is there an almost 
desperate need for more efficient means of presenting 
the subject. This same trend is also developing in the 
field of American history as we attempt to add more 
information or to delve deeper into the subject at all 
levels. 

The social studies also has another obstacle not usu- 
ally present in other areas. Since it is not (or should 
not be) a purely factual subject, it is almost impossible 
to hand the student a history textbook and expect him 
to appear the next day with anything resembling or- 
ganized complex thought. Even with a well organ- 
ized text, the problems of varied reading ability and 
the inability to extract only the important material from 
the context because of the lack of basic repertoire in 
the area lead to defeat in many cases.- As a result, 
reading assignments from the text are often challenges 
requiring the student to learn from the text those asso- 
ciations he thinks the instructor thinks the author be- 



lieves are important. Often the student is completely 
incapable of doing so. 

Because of this situation, much of the time in the 
conventional social studies classroom is spent in sorting 
out, exploring, and explaining the material found in the 
book. This, of course, leaves little time for other 
things, particularly for the study in depth of the great 
people and events of history. Perhaps this dehumaniza- 
tion of the subject is the reason for the apparent lack 
of interest on the part of the students for this area of 
their studies. History is people; without them, there is 
no history, yet students often perceive history as having 
no “human history.” 

In recent years an attempt has been made to return 
the great people and events to the history books by 
following only the important developments of western 
civilizatiou from beginning to end with only a mini- 
mum of attention being given to factual material. This 
method can be called the “concept school,” as opposed 
to the “facts school” which only skims the factual 
surface of history. Neither the “facts method,” which 
leaves out people, nor the “concepts method,” which 
ignores facts, is a satisfactory solution to the problem 
of cramming five thousand years of history into the 
minds of our young people in one pitifully short school 
year. 

Perhaps a well-developed program could offer relief, 
or even a solution, for this dilemma. Most educators 
recognize that both facts and concepts (i.e., critical 
complex thinking) are important in the field of social 
studies. Most will agree, however, that it is very diffi- 
cult to obtain one without the other. In the sciences how 
can complex thought be developed with reliance upon 
facts? This seemingly presents quite a challenge to the 
medium of programmed instruction, and it is this chal- 
lenge of teaching both facts and concepts that most 
programmers have avoided. Admittedly, it is an almost 
impossible task, particularly since it is rather difficult 
to find two historians or even two teachers who agree 
on all the important concepts of history. 

But ours is not the task of solving the world’s prob- 
lems or even the problem of those who teach social 
studies. In fact, ours is not even the problem of telling 
the teacher what or how to teach, much less attempting 
to replace the teacher. Strangely enough, this may be 
the key to the solution of the problem. 
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If a program could be developed to present one side 
of the social studies story so well that the teacher need 
spend very little time on the material in this area, then 
this precious time could be spent in developing the 
other area effectively. At this point let us will the con- 
troversial and somewhat fluid area of “concepts” back 
to those brave enough to teach the subject, and from 
henceforth deal with the “easy” area of history, the 
facts. In this manner we have created a medium more 
familiar to the programmer and most certainly one he 
is quite capable of handling. Admittedly, we have side- 
stepped a major challenge, to produce a complete pro- 
gram in world history, but this can wait for future 
development and growth in the field of programmed 
instruction. Our immediate problem is to produce a 
package that will enable the classroom teacher to do 
an effective and efficient job of teaching a complete 
course in history. This can be accomplished by remov- 
ing the burden of teaching factual material, thereby 
enabling her to devote class time to the development 
of concepts and critical thinking in her students. Thus 
the main objective of a program in world history is to 
present to the student a solid foundation of factual 
material, a basic repertoire, on which a teacher can 
build an interesting, detailed, and challenging study of 
world history. This type of program can be called a 
foundation program, since its sole purpose is to pro- 
vide a solid foundation on which a teacher can build 
a complete course. Like a building, its final height 
depends upon the goals of the designer as well as on 
the finnness of the foundation. 

The purpose of this program in world history is to 
teach the student in as short a time as possible the facts 
of history so that he can utilize this knowledge in the 
classroom under the guidance of a capable teacher. 
In the program, therefore, we are concerned with such 
factors as the location, names, dates, persons, events, 
causes, effects, and contributions of the major civiliza- 
tions of the world. 

This type of material lends itself quite readily to the 
Skinnerian, or linear, format of programming. The 
major problem involved is that of introducing new 
material, since, unlike mathematics or science, there 
is little basic repertoire or knowledge that can be ex- 
panded upon. To introduce this new material without 
overloading the program with copy frames, extensive 
use was made of display panels consisting of pictures, 
charts, or time lines. From these, the student can 
obtain, with a reasonable amount of searching and 
thinking , the necessary information to complete the 
frames. 

Other than the extensive use of panels, there is very 
little difference from the conventional type of linear 
program. The frames are brief, in most cases, and 



require an overt (written) response on the part of the 
student. On the trial runs the responses are written 
directly on the program but of course this will be 
changed in the future to make the package reusable. 

The standard techniques of review and criterion test 
fraraes have been used to check on the progress of the 
student. The program also incorporates, with varying 
degrees of success, the use of wash-back or review 
techniques for the slower student. One problem in this 
area is the conservation of frames. With a program of 
this scope, the number of frames could easily get out 
of hand and begin to climb toward rather impractical 
numbers. 

Post-tests are given at the end of each chapter, and, 
in the cases of very long chapters, at logical subdivi- 
sions within the chapter (at approximately two hundred 
frames). The post-tests are of the objective type, re- 
quiring written recall responses. 

The program in world history is being tested on two 
groups of ninth grade students at John Nance Gamer 
Junior High School, Northeast Independent School Dis- 
trict, San Antonio, Texas. This course is generally a 
tenth grade course, but is being offered at the junior 
level for the academically talented student. Listed 
below are some general characteristics of the two 
groups. 









Mean 


Median 






Size 


Age 


IQ 


IQ 


Range 


Group 1 


30 


13-14 


120 


121 


98-145 


Group 2 


28 


13-14 


122 


120 


103-143 


Both the 


test groups and 


several control groups of 



the same caliber were given a standardized pre-test at 
the beginning of the year. This same test will be 
administered at the end of the year to see if there is 
any significant improvement by the test groups over 
the control groups taught by the conventional method. 

At the present time it is impossible to arrive at any 
final conclusions as to the results of the program since 
it has not been completed. The most that can be said 
is that the technique does work. The students have 
foimd the program interesting and in some ways chal- 
lenging, and have continued to work on the program 
with no pressure from the teacher. In fact, some of 
them have taken more of an interest in programming 
than they have in the subject itself. 

Of major concern was the fact that the program was 
being tested for the first time on rather talented stu- 
dents. There is a possibility that such students could 
do well on any type of material, thus making invalid 
any test results derived from these students. To combat 
this problem, the slower students of both groups, who 
approach the national norm, were carefully observed. 
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The chart below is a comparison of the grade averages 
of the top five and the bottom five students from Test 
Group 1. 







Average: 


Average: 






6 Test Grades 


6 Test Grades 


Student 


IQ 


Frames 1-500 


Frames 501-130 


1 


145 


97 


93 


2 


139 


96 


92 


3 


136 


95 


87 


4 


135 


95 


82 


5 


132 


98 


85 


Average, Top Five 96 


88 


26 


108 


88 


82 


27 


106 


96 


95 


28 


104 


98 


88 


29 


100 


87 


90 


30 


98 


83 


50 



Average, Bottom Five 90 81 



There are differences in their grade averages. But with 
the exception of student #30, who dropped the course 
at mid-term, the slower students showed a gradual im- 
provement in their grades even thou^ the material 
became more difficult. 

Admittedly, the grades aro not as high as would be 
desired since they do not attain the goal of ninety per 
cent of the students Imowing ninety per cent of the 
material. However, on some of the post-tests this goal 
was attained, and on others the results indicated that 
the goal probably could be attained without drastic 
revision of that section of the program. At the present 
time there is one section that probably needs rather 
extensive revision. 



Even though there are no final results upon which 
to judge the results of the program in world history, 
it is possible to arrive at several general conclusions: 

1. The idea of a foundation program does work. 
With the test groups, not as much free time has been 
attained as was first considered possible, but this was 
due to the time involved by the iostructor’s having to 
research, write, type, and mimeograph approximately 
260 pages of material. A prepared program would 
allow the classroom teacher relatively large amounts 
of classroom time to explore the subject in detail. 

2. The novelty effect of the programmed material 
does not seem to diminish to any great extent because 
of several factors: 

a. Constant high grades are encouraging to the stu- 
dent. 

b. Logical and careful sequencing of the material 
makes the information easy to learn. 

c. More class time for discussion and research make 
the subject more interesting. 

3. The completion of a set of frames for homework 
leaves the student with a sense of accomplishment of 
actually having learned something, an experience rarely 
found with textbook assignments. 

4. Spelling accuracy generally improved because of 
constant repetition of important words. 

5. The slower learner does not become lost and dis- 
couraged as the material becomes more complex and 
detailed, but actually improves with time due to the 
highly organized nature of the program. 

6. Programming a course in world history is no easy 
task. But for a teacher concerned with the problems 
of helping students learn, it is a rewarding one. 
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PROGRAMS IN THE LANGUAGE ARTS— AVAILABLE AND DESIRABLE 

F. Alien Briggs and Robert K. Branson 

This discussion between F. Allen Briggs and Robert K. Branson brings out the diffi- 
culties often encountered between teacher and programmer. Mr. Briggs suggests that 
specific skills in language arts instruction can be programmed. He points out^ however^ 
that in many areas there is disagreement about “correct" answers which will hinder 
programmers. He argues for programs that really teach as opposed to programs 
that are merely exercise books. Mr. Branson^ speaking for the programmer^ wants 
clearly defined objectives from which to program language arts. He feels that pro- 
grams in the areas of appreciation and reasoning will have to be written and tested 
to prove that such programming is possible. 



Mr. Briggs: Language Arts is an area of study com- 
posed of many specific skills, the application of which 
produce efficient language usage. Not only may the 
field be divided into the traditional parts of reading, 
writing, speaking, and listening, but under these general 
divisions there are such specifics as spelling, grammar, 
punctuation, vocabulary, paragraph form, rhetoric of 
different forms of larger units, the attribution of 
sources, reading skills, imagic devices, and many others. 
Because so many different things are taught in the lan- 
guage arts classroom, there is a great deal of room for 
auto-instruction. 

Although these different skills may be mutually sup- 
porting, there are many areas which may be studied in 
a varied sequence. Different from the discipline of 
mathematics where addition and subtraction skills are 
necessary before multiplication and division can be 
efficient, language arts skills are not so rigidly sequen- 
tial. The breadth of choice makes it possible for a 
teacher to select an area for programmed teaching 
which may be used with most of the members of a 
given class. On the other hand, there is a sequential 
feature within most of the skills; some of the grammar 
skills should be learned first, and later in the school 
experience of the child, more advanced grammar skills 
are in order. 

Mr. Branson: Although it may be very desirable 
to think of utilizing the strengths of programmed learn- 
ing in the language arts area, there are probably other 
changes which would benefit the field to a greater 
degree, i.e., if every teacher of every subject from grade 
school through the university would insist on his stu- 
dents writing in the English language and not accept 
his materials written in only an approximation of 
English, then symposia, panel discussions, and many 
controversies in the field of language arts would be 
essentially eliminated because the negative aspects of 
language teaching in the language arts area would have 
been eliminated. 

Mr. Briggs: There are certain generally recognized 
limitations of language arts teaching, and many of 
these limitations are complemented by the strengths 



of programmed instruction. In the first place, most 
school children are expected to learn the skills assigned 
to a given year although they have not mastered the 
skills taught the previous year or years. At this point, 
progr ammin g’s emphasis upon mastery of a skill before 
the student goes ahead to the next level will free the 
student from lock-step enforced inability to learn. In 
the second place, the frequent review and reteaching 
made necessary by incomplete mastery at all levels 
often forces a student who has done his work to repeat 
something in which he is not interested. Programming’s 
ability to allow each individual to work at his own 
rate and at his own level will prevent the boredom (and 
consequent disciplinary problems) of needless repeti- 
tion. 

In language arts, as much as in any other subject, 
the partially acquired skill is of minimum value; pro- 
gramming should allow emphasis upon fundamentals 
until maximum proficiency in the minimums has been 
attained. 

Mr. Branson: Programming like any other method 
of teaching in the language arts area suffers from a 
certain limitation. Perhaps we ought to decide some- 
time if we are going to teach “about English’’ or “how 
to use English.” Tfie terminal behavior in either case 
would be different. There is considerable doubt that 
teaching about English will improve one’s ability to 
use the language. With few exceptions, most of us 
speak the language far better than we write it. I agree 
with those who contend that this occurs because we 
learn to speak the language without ever going through 
a procedure of talking about the language. 

Mr. Briggs: Although there is general agreement 
about desirables, there is little agreement among “ex- 
perts” in language arts about how the attained skills 
may be measured and even less common ground about 
the correct path by which the objectives may be at- 
tained. Most teachers agree that the child should read, 
write, speak, and listen expressively and efficiently; all 
of them try to help their students do better. 

On the other hand, methods by which these skills 
can be evaluated are violently questioned, and even 
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greater disagreement exists concerning the level of per- 
formance which may be expected at the different levels 
of schooling. In a number of the sub-disciplines, there 
is almost no agreement as to the most effective path 
toward proficiency. In grammar the battle between 
usage, prescriptive and linguistic proponents wages so 
high that no program could satisfy all three, and the 
“separated” as opposed to the “fused” methods of 
teaching often cause real quarrels in language arts 
circles. 

There have been a number of studies made which 
give some indication of the grade at which certain 
skills should be taught, but these studies have not be- 
come effective at the level of the curriculum of the indi- 
vidual school. There are still many teachers who 
attempt to teach the children all they know at any grade 
placement. 

A fourth difficulty for the programmer is the diffi- 
culty of ascertaining the “correct” answer in many 
language situations. In some, such as spelling or the 
agreement of verb and subject, the correct answer is 
relatively unquestionable and may be used as a con- 
firmation in the usual sense. In others, there are a 
number of answers which may have relative degrees of 
“correctness” and the use of one of them as a confirma- 
tion (even if it is understood that another answer 
“which says the same thing” or “which accomplishes 
the same end through different means”) will result in a 
lack of freedom sufficient to make the attainment of 
goals of individuality impossible. 

Many of the programs now on the market seem to 
have confused the knowing how to do with the ability 
to do; they offer statements of principle and drill in 
“exercises”; no more than in the least effective conven- 
tional instruction do they provide for transfer into the 
student’s own writing and speaking. In point of fact, 
many of the programs now on the market are really 
nothing except workbooks, chopped up into frames, 
and with the teacher’s manual or key also supplied in 
bits. 

In using the new device of programming in language 
arts, there seems little advantage, other than commer- 
cial, in perpetuating the faults of the present system. 
The new wine of individual pacing and confirmation 
may well serve to burst the old wine skins of busywork 
with a minimum of learning. 

Mr. Branson: Professor Briggs has made a very good 
point in elaborating some of the shortcomings of pro- 
grams on the market which deal with the problem of 
the desirable terminal behavior for the student. So 
far no one has written a program that can establish 
“creativity,” “ori^nality,” or other highly desirable fea- 
tures. On the other hand, some have vastly improved 
their ability to be creative and original by using pro- 



grams available on the market. Some students have 
been able to transfer behavior established by the pro- 
gram. Others have learned what is in the program, but 
have not shown improvement in their ability to write 
the language. Another problem that the programmer 
faces in developing material for us in language arts is 
that of defining acceptable terminal behavior. 

Mr. Briggs: Some real attention might well be given 
to the presence of multiple language levels, i.e., one 
writes differently than he speaks — sight recognizes 
words which he never hears, and is impressed by dif- 
ferent rhetorical devices as he listens than when he 
reads. Some recognition of the multi-level situation in 
language would be realistic and helpful. 

Long, complete courses in grammar, punctuation, or 
vocabulary imply the pressure “to finish the book,” or 
“to go through the course from A to Z.” Such a situa- 
tion is a maximum hindrance to major advantages of 
progr ammin g which are capitalizing on the student’s 
interest and staying on the subject until the skill is 
mastered. Of greater benefit would be a series of at 
least semi-independent units which might be used in 
any order best suited to the individual situation. Susan 
Markle’s program in vocabulary is a helpful step in 
this direction. In addition, there might well be parallel 
units, at different grade levels, so that a given skill can 
be taught at the student’s point of maturation. 

Greater experimentation with and understanding of 
the “multiple-right-answer” technique seems useful in 
language arts. Some of these answers might well be 
linked into a line through such a device as “but have 
you also thought that maybe” to destroy the one ques- 
tion — one answer syndrome. 

Mr. Branson: We could conceivably write a techni- 
cally perfect program that would be utterly useless 
because the behavior established by the program would 
not be beneficial to the student. Granted, many techni- 
cal problems have not been solved. Nevertheless, the 
emphasis must be placed on the specification of the 
objectives and the elaboration of the purpose of the 
program. Many people argue that poetry, literature, 
music and art appreciation should not be programmed. 
I have no argument with those who hold this position. 
If a programmer places a program in front of me en- 
titled “Music Appreciation” or “Art Appreciation” and 
the program achieves its objectives, then, and only then, 
can we argue that art appreciation “should” be pro- 
grammed. When someone argues that a given subject 
matter cannot be programmed, then it is time to say 
that only when material is successfully programmed 
will this argument be overcome. 

Mr. Briggs: A branched, linear device might well be 
helpful so that a student could skip skills which he 
already knows and go at once to other skills. In an 
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“understanding of poetry” program, it is possible to say 
that reading and comprehending comes at three levels: 
(1) understanding what the author says, (2) under- 
standing what the author implies, and (3) being able 
to generalize from the specific implication. A program 
such as this might provide the selection to be read, 
followed by some questions, six or more. If the student 
got none right, he would do the exercises on under- 
standing the text; if he got part right, he might go on 
to implication or application. 

There are a number of areas in language arts which 
have hardly been touched by programs. Oddly enough, 
punctuation programs are scarce while spelling and 
grammar programs are fairly well represented. A word 
of warning might be said to teachers seeking a gram- 
mar program. The entire program should be read 
(worked) by the teacher before it is selected; some 
of the information presented is dead wrong, and in 
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other cases, the writer of the program may use a phi- 
losophy of gr amm ar completely unwelcome to a given 
teacher. There are some good programs in vocabulary, 
but there are none in recognizing devices of collective 
sanction (propaganda) or in reading for larger units of 
expression. There seems to be no programs in the 
rhetorical devices of larger units (paragraphs, essays, 
etc.); there is nothing in phonetics and phonemics or in 
most of the areas commonly covered in speech (except 
debate). 

Mr. Branson: In summary, it appears that there are 
many problems in the language arts area which are the 
sole responsibility of subject matter specialists. It is 
essential that we determine the desired terminal be^ 
havior for any program in this field, then set out to 
achieve that behavior. Until we can arrive at mutually 
acceptable goals, we cannot agree on what course we 
should follow. 



SOME PITFALLS IN THE CLASSROOM USE OF PROGRAMS 

Robeit K. Branson 

Robert Branson cites the dilemma posed by the fact that a teacher’s first use of 
programmed material probably will be for remedial or enrichment purposes, whereas 
most programs were not developed for these uses. He also cites experimental data 
that raise questions about the traditional view that cheating is detrimental to learning. 



The purpose of this paper is to elaborate some of 
the problems that teachers will face as they attempt 
to integrate programmed materials into their class- 
rooms. Those programs that are introduced will most 
likely be used as supplementary, remedial, or enrich- 
ment devices. Although programs should be effective 
when used in these ways, most programs have not 
been planned for these uses. 

Consequently, the initial problem in the use of pro- 
granuned materials in a classroom may well be an 
inappropriate application of the program. One of the 
more important reasons for such a situation is that 
there are not enough programs available to meet every- 
one’s specific needs. There is probably little that can 
be done to reduce this hazard other than to suggest 
to program users that they attempt to match the in- 
tended use of the program with its objectives. 

If , in a program on FRACTIONS, the programmer 
has assumed the abilties of addition, subtraction, multi- 
plication, and division, he may then wish to proceed 
as quickly as possible into the manipulation of frac- 
tions, without reviewing these fundamental operations. 
It is true, however, that there are vast differences in 
the abilities of students to add, subtract, multiply, and 
divide. Perhaps a short test, supplied by the program 
maker would help the teacher determine if the student 
had all the prerequisites for taking the program. In 
the General Programmed Teaching Corporation 
(GPTC) testing laboratory, we have generally found 
that students taking programs without the correct pre- 
requiates always have considerably more trouble than 
those students who are proficient prior to starting the 
program. An effective trial of a program cannot be 
assured unless all of the specified conditions called for 
by the program are satisfied. 

Although I hesitate to mention this next point, since 
it has become a family skeleton in programming tech- 
nology, cheating still remaius a serious problem. Theory 
demands that students respond prior to checking the 
answer given in any frame. Presumably, the response 
should occur prior to the “reinforcement” provided by 
the confirmation. 

During a recent program administration in a private 
school in Albuquerque, we encountered a wide-spread 
cheating problem. We were testing a program on multi- 
plication and division in a third grade class. We first 



tried using programs with masks, and found that stu- 
dents virtually always checked the answer before re- 
sponding. We then decided to provide machines for 
each student. The machines, Min-Max IPs, were used 
for the duration of the program. Our reproduction 
paper did not work well in the machines, and our frame 
layout permitted the student to see the answer in some 
cases before actually making the response. Our direc- 
tor of testing and the classroom teacher were both up- 
set by the cheating. It reflected badly on the teacher 
whose students were cheating so persistently and so 
effectively. There are a variety of ways that one can 
detect cheating in the use of programmed materials, 
but the simplest seems to be the direct observation of 
the subject. When the program had been completed by 
the students, the data were analyzed to see if any 
measurable effects had been demonstrated. 

The first thing we discovered after administering the 
post-test was a substantial percentage gain for each of 
the students. A second rather startling fact was that 
there were no reliable correlation coefficients between 
the estimated amount of cheating and the post-test 
achievement score. Neither could we find any reliable 
correlations between intelligence and achievement test 
scores, pre-aud post-test errors, or response errors and 
post-test scores. There were also no reliable differences 
between scores of students taking the program in the 
classroom, and those taking the program in the GPTC 
testing laboratory. The students in the laboratory were 
under constant surveillance in order to prevent cheat- 
ing. 

Although cheating seems to be a serious problem 
in programming (and one that is very likely to be en- 
countered in the classroom where a teacher must 
monitor some twenty to forty students) with our mea- 
sures of achievement, we have found no evidence which 
suggests that cheating is detrimental to learning. We 
will grant that this is a startling conclusion, and that 
it is only tentative; nevertheless, until we have more 
exhaustive data, it appears that we must re-examine our 
assertions about the theoretical effects of cheating. 

The next situation that I would like to discuss oc- 
curred at Louisiana State University in the summer of 
1961. A current and serious problem facing many 
universities is what to do with those students who do 
not have adequate background to do university work. 
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Some schools have failed these students while others 
have established elaborate remedial curricula in an at- 
tempt to raise the students to university level. Recent- 
ly, there has been a trend away from the remedial 
classes and back toward the “sink or swim” philosophy 
prevalent some years ago. The mathematics depart- 
ment agreed to try programmed materials in an attempt 
to teach many otherwise good students enough about 
mathematics to meet departmental standards. 

Two sections of a remedial mathematics class were 
selected and programs were purchased for those stu- 
dents. The students were told that as soon as they 
could pass the final examination, they would be allowed 
to enroll in a regular credit course. A monitor was 
provided to answer the students’ questions. Although 
some students progressed through the program, passed 
the final examination, and went on to university level 
courses, the majority of the students did not pursue 
the programs actively and many of them failed the 
final examination. Interestingly enough, there were no 
significant dMerences in the number of students failing 
the final examination in the regular remedial classes 
and those in the programmed classes. Some students 
in the remedial classes finished the programs early and 
were then free for the remainder of the semester. One 
student finished in two weeks and was permitted to 
enroll in a regular credit course which he passed with 
an adequate grade. 



The problem in this case seems to be one of moti- 
vation. We all know that with programmed instruc- 
tion, learning can be fun. It is quite possible that 
these students were so far behind and had so little in- 
terest in pursuing a university degree that they made no 
attempt to apply themselves to the programs. It is also 
true that these programs were not intended to be 
remedial materials. And, although the programs did 
at least as well as regular classroom instruction, thus 
relieving the department of a certain financial burden, 
the programs did not accomplish what it was hoped 
that they might, which was to bring all of the students 
up to an acceptable university level. 

This is not an indictment of programming, per se, 
it is merely a description of a situation in which pro- 
grams were used in an attempt to accomplish results 
that teachers had failed to accomplish. The findings 
were not dramatic. 

Arthur Melton once remarked that programmed in- 
struction would never achieve its ultimate effectiveness 
until a complete systems analysis was made of educa- 
tion. When such information is available, we will 
know when, where, and how to integrate programs 
into the system. Only then can we expect to avoid 
large pitfalls in the uces of programmed instructional 
materials. Conversely, we can also see that many of 
our attempts to use programs have been highly success- 
ful in spite of our making errors in using them. 
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WHAT SERVICES TEACHERS AND EDUCATORS WANT AND NEED 
FROM PROGRAMMED LEARNING EXPERTS 

Russell Robbins 

Russell Robbins feels fhof programmed instrucHon and automated teaching devices 
will play an exciting and expanding role in the future. How rapid and how extensive 
these new methods of instruction will be implemented is not known. Because sufficient, 
valid, and reliable research evidence is at present unavailable, many teachers are 
apprehensive about the effect that programmed instruction might have on the teaching- 
learning process, the students, the whole school organization, and on the role of the 
teacher. Although the prospects for programmed learning appear promising, many 
questions remain to be answered. 



A. number of educators and others believe that pro- 
grammed instruction, used with or without teaching 
machines, is the answer to many of the problems in 
education. Enthusiastic proponents of programmed- 
instruction write in glowing terms of the superiority 
of this method over methods used in schools today. 
Their enthusiastic viewpoints, however, are not shared 
by all teachers and educators. One finds a certain 
reluctance on the part of many to accept autoinstruc- 
tion as the salvation of public education. The reluct- 
ance on the part of some teachers to accept and whole- 
heartedly support programmed instruction probably 
stems from several viewpoints. 

First, although critics admit that there is some favor- 
able research evidence to support the superiority of 
programmed instruction versus more conventional 
teaching methods in some areas of the curriculum, there 
is insufficient data, at this time, to draw final and valid 
conclusions. After reviewing research studies on auto- 
mated learning one writer (Stolurow, 1962) stated: 

While there has been a history of research studies relating 
to automated instruction, they have only scratched tlie surface. 
Despite their number, the research studies relative to auto- 
mated instruction are still limited; the resultant findings are 
more provocative than definitive. 

Second, some teachers are reluctant to give support 
to programmed learning because they fear teaching 
machines might replace them. Proponents emphatically 
deny that teachers will be replaced by machines. How- 
ever, one can only conjecture on this matter because 
the roles of teachers and administrators presently lack 
clear definition. In addition to fears of being replaced, 
some teachers also feel insecure about the whole con- 
cept of programmed learning. Their insecurity may 
come from a lack of knowledge of auto-instructional 
devices and how they will be utilized in the instruc- 
tional program. 

In order to allay teachers’ fears and secure their 
support for programmed learning, it is imperative that 
many questions be answered regarding the role of the 
teacher and administrator. 

Generally spealdng, writers on the subject of pro- 
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grammed instruction are prone to touch ligihtly on the 
role of the teacher using this new medium. Pressey has 
stated that automatic devices would free the teacher 
for her most important work of developing in her pupils 
fine enthusiasms, clear thinking, and high ideals. 
(Pressey, 1926). Barlow views teaching machines as 
a way of implementing and increasing the teacher’s 
instructional effectiveness. (Barlow, 1960). Much of 
the drudgery may be taken out of the teacher’s job to 
the extent of eliminating much of the detail of in- 
struction and drill. Although much of the drudgery 
may be eliminated, the teacher will still continue to be 
responsible for the pupil’s instructions. The teacher 
will thus have more time to spend on planning for 
individual needs, to better master her own subject 
matter, and to become better acquainted with her 
students (Stolurow, 1962). Conjecturing on the revolu- 
tion that will take place in education, Stolurow stated 
that high school teaching staffs will include educational 
analysts who will have specialized in various subjects, 
and who will review student records on an individual 
basis (Stolurow, 1962). Teachers will be assigned a 
minimum of routine duties. Besides constantly seek- 
ing to discover problems and needs for special atten- 
tion, teachers will diagnose information and difficulties 
and make recommendations for improvement. The 
concept of the role of the teacher wiU be changed en- 
tirely to that of a thinker and planner. In the coming 
era of auto-instruction and teaching machines, Ramo 
(1962) sees the teacher as a type of educational spe- 
cialist; a teaching engineer not only concerned with the 
educational process, but with the design of the machines 
and programming of instructional materials. 

Few people doubt that the role of the teacher will 
change in the future because of teaching machines 
and other methods of auto-instruction. What these 
changes will be are mere speculation at this time, and 
this uncertainty has . aturally created misgivings on the 
part of many teachers. Although some universities and 
school systems are experimenting with programmed 
learning and its implications for instruction, many 
teachers throughout the country are not well informed 
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on the subject. Other teachers question the validity 
of extravagant claims made by some proponents of 
programmed instruction, while many others are con- 
cerned with the changes that will occur relative to their 
roles as teachers. 

During the second semester, 1963, a group of senior 
student teachers from Northern Illinois University who 
were finishing their teacher training programs in June 
were asked to pose some questions concerning pro- 
grammed instruction. It was assumed that their train- 
ing was comparable to that received in most universi- 
ties and that they were probably as well acquainted 
with audio-visual devices as were most new teacher 
training graduates. 

The questions posed by the students were grouped 
into two general categories: (1) the effect programmed 
instruction might have on children and on the teach- 
ing-learning process; and (2) the effect programmed 
instruction might have upon the curriculum and the 
teacher. A large portion of the questions seemed to 
indicate a feeling of concern on the effect programmed 
instruction might have on children and the teaching- 
learning process. Students wondered if the emotional 
and social growth of pupils might not suffer by the 
preponderant use of such devices. Is it possible to 
program materials which will teach democratic values, 
restructure attitudes, and stimulate creativity and re- 
flective thinking? A brief resume of the questions in 
this category are: (1) Will the use of teaching machines 
hinder emotional and social growth?; (2) Will the use 
of teaching machines and programmed materials hinder 
creativity?; (3) Can programmed learning teach values 
and attitudes?; (4) How can teaching machines moti- 
vate and hold childrens’ interests at a high level?; 
(5) What happens when the novelty of working a 
machine wears off?; (6) To what extent will programs 
provide for individual interests of the child?; (7) Does 
programmed learning allow for reflective thinking?; 
(8) Does programmed learning allow for equally 
effective opportunities for all levels of learners?; (9) 
Does programmed learning permit children to discover 
for themselves?; (10) Can children benefit from dif- 
ferent ideas and viewpoints of others in the room?; 
(11) Is it possible that the use of teaching machines 
might lead to rote learning rather than to meaningful 
learning?; and (12) How will programming affect 
teacher-pupil planning? 

The remainder of the questions appeared to be 
m ainly concerned with the effect programmed instruc- 
tion might have on teachers and upon the curriculum 
Students were interested in finding answers to these 
additional questions: (1) Will teachers be able to 
develop programs themselves or will tliis be the job 
of programming specialists?; (2) If programs are de- 



veloped by programming specialists, how wiU the ob- 
jectives of individual teachers be incorporated in the 
program?; (3) Can all subjects taught in the ele- 
mentary and secondary school curriculum be pro- 
grammed? If not, what subjects do the experts believe 
can best be programmed?; (4) Will each teacher have 
the opportunity to choose the programs she will use?; 
(5) Will teaching manuals or guides be available with 
programs?; and (6) What preparation is necessary 
for a teacher to be able to effectively program material? 

Although some of these questions may seem naive 
to the expert in the field of programmed instruction, 
many may be apropos to the classroom teacher. Ques- 
tions such as these must be answered if the role the 
teacher will play in instruction in the future is to be 
clarified. There are also many problems confronting 
school administrators who wish to incorporate pro- 
grammed materials in their schools today. 

The problem of school administrators have become 
complex, serious, and challenging. A number of their 
problems have been brought about by the introduction 
of new media such as auto-instruction. Administrators 
face the same dilemma as do teachers in that many of 
them have had little or no training or experience with 
programmed instruction. How does an administrator 
decide what programmed materials to purchase? In a 
large school system where trained research personnel 
are available, the system can conduct its own research 
on automated devices and materials. This does not 
solve the problem in the small school district, however, 
where research personnel are not available. The small 
district administrator must obtain programmed ma- 
terials from a commerical company or wait until re- 
search programs in larger school districts or universities 
have developed fully tested programs. If an adminis- 
trator decides to purchase materials from commercial 
companies, he must be able to determine the merits 
of each program under consideration. To make a de- 
cision as to what program or programs would best fit 
his district’s needs, the administrator should have a set 
of criteria for evaluating programs. Several sets of 
criteria have been devised, but none have yet achieved 
wide acceptance. 

Auto-instructional advances have confronted the ad- 
ministrator with many other kinds of problems. How 
will teaching machines and programmed instruction 
affect the curriculum, the staff, the physical plant? 

Team teaching, auto-instruction materials and equip- 
ment will undoubtedly require changes in the curricu- 
lum, in the qualifications of the staff, and in the physi- 
cal plant. Scheduling patterns will change and new 
patterns will have to be developed. These are but a 
few of the problems that administrators will have to 
consider with the coming of auto-instruction. 
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A PROGRAMMED TEXT IN PROSODY 

Miriam W. Hiesfer 

An exhibit of teaching machines led Miriam Hiester to develop a three part pro- 
gram on prosody for use in her college English classes. In completing her successful 
program she found that her conventional instruction was also improved by applying 
the rules of good programming. She outlines a bright future for college level pro- 
grams and encourages teachers to try programming. 



The need for writing a programmed text on prosody 
emerged from the fotiv wing conditions, all very famili- 
ar to the teacher .T f eshman college English. Few 
students in freshm.ai English know the basic principles 
of prosody. The material given on this subject in the 
average textbook is very scanty. Library reference 
books are too detailed, too lengthy, too technical for 
students on this level. Presentation of the material by 
lecture with illustrations on the chalkboard is time- 
consuming and lacking in effectiveness in many re- 
spects: many students take poor notes, misspell terms, 
become confused; very few make good notes or score 
well on tests involving use of terms and procedures. 

The way to improve this situation became apparent 
when I heard about programming and investigated its 
possibilities. At first I thought that only grammar, 
spelling, punctuation, and such other mechanical, right 
or wrong-material could be programmed. In 1960, I 
attended a lecture, at the University of Texas, on 
teaching machines and observed a machine in opera- 
tion. One of the programs was an analysis of a poem 
by Gerard Manley Hopkins, for which they handed 
out mimeos. Trying out this mimeographed program 
on my students in the San Antonio College, I found 
that its measurement of their ability to analyze a poem 
correlated very closely with my own estimate gained 
by the usual means: tests and class participation. Final 
motivation to try writing a programmed textbook came 
with information gained from lectures given in the 
Medical Field Service School by Captain David Wark. 
He had specialized in programming in his graduate 
work and was encouraging faculty members in MFSS 
to investigate possibilities of using this method in the 
school. 

The procedure I used is very simple. First, I listed 
the various topics of prosody and decided in what 
order the students could best learn them. I then broke 
each topic down into small parts, progressive steps. I 
used the linear method, placing an explanation of one 
small part followed by a question on page, 5 x SV 2 
inches; the next page contained not only the answer to 
the question on the preceding page, but also the next 
question or the next explanation and question. I did 
this to save space, even though I realized that some 
of the experts might disagree. Another deviation from 



reco mm ended procedure is that I included more infor- 
mation on each page than is usually found in the so- 
called scientifically constructed commercial texts. Since 
these were college level students, I felt they could take 
larger “steps” without adverse effects. 

The first part of the program contains a definition 
of rhythm, an explanation of stressed and unstressed 
syllables, names of the kinds of metrical feet and terms 
for lines according to number of feet. The second part 
consists of an explanation of rime and rime schemes 
and included various often-used stanza forms: couplet, 
tercet, quatrain, ballad, sonnet. The third part con- 
tains poetic devices: alliteration, onamatopoeia, sunile, 
metaphor, personification, etc. 

Believing that the students would learn better if they 
made active responses, I prepared a separate answer 
sheet, keyed to the questions in the text. The students 
responded by filling in the blanks on the answer sheet, 
not on the program. The separate answer sheet has 
two advantages: first, programs are not used up by 
each student, making it possible for him to go through 
the text as many times as he wishes; and second, the 
texts could be collected at the end of the course and 
used for another class. Most students requested one or 
two additional answer sheets. They were allowed as 
many as they needed. 

Periods in the Evening Division of the San Antonio 
College are 3 hours with a 20 minute break near the 
middle. I used the last half of one period for the 
programmed instruction. First, I explained the purpose 
of the booklet, and how the students were to use it. 
Then I handed out the booklets and answer sheets, 
stressing that answers were to be recorded on answer 
sheets, not in the booklets. 

There was a dead silence for 45 minutes; then pairs 
or small groups of students began discussion points. 
I moved among the “discussion groups” to answer 
questions and note progress. All students went through 
the booklet at least once during the period; some twice. 

Assignment for the next period was to read some 
selected poems, from the textbook, and apply princi- 
ples learned. 

The programmed text has now been used with three 
classes. Students in all three classes were most en- 
thusiastic and responded well. Many asked to keep 
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the book after the final examination “for future refer- 
ence.” Students receiving instruction in this manner 
scored significantly higher on prosody portion of test 
than those receiving the instruction by the lecture 
method. 

The following semester, I had some of the students 
in the sophomore En^sh (Survey of English Litera- 
ture) course and found that they had retained the 
information weU. One who had not been in my fresh- 
man class requested “one of those books your other 
students had that taught them so much about poetry”. 

The programmed textbook seemed to stimulate in- 
terest in poetry itself and the interpretation of it. Two 
students (boys!) from separate classes wrote poems 
and handed them in for my appraisal. 

The programmed text injected a certain amount of 
excitement into an otherwise routine explanation. My 
enthusiasm grew as I progressed in constructing the 
program. Some of this enthusiasm carried over to the 
students. In fact, a teacher must have a positive atti- 
tude toward a program to make it effective. 

Programming this small part of a course had its effect 
upon other units I taught. I became more aware of 
step-by-step instruction progressing logically toward a 
desired objective. The state of min d induced by pro- 
gramming, a sort-of “scientific” point of view of sub- 
ject matter forces a teacher to search for order in the 
materials themselves and place this above any personal 
preference or eccentricity. Many English teachers fail 
to see the order inherent in literature. You are never 
quite the same again after programming a course or 
even a very small part of it. Teaching literature be- 
comes less a private than a professional matter. 

In College English, January 1962, Kenneth S. Roth- 
weU published an article entitled “Programmed Learn- 
ing; a Backdoor to Empiricism in English Studies.” He 
states. 

The full implications of teaching machines and programmed 
learning to the teaching of English composition and literature 



have not, so far as I know, been completely thought out. . . . 
How much or how little of English studies can effectively be 
programmed remains a question which only experimentation 
can answer, though the likelihood is that wide-scale studies 
may open up more possibilties than anyone can now envision. 
Virtually all remedial English at college level could be handled 
by automation. Teachers, relieved of the executioner's role, 
could then become counselors rather than taskmasters. Their 
human talents, at a time when trained talent is scarce, could 
be salvaged for situations better suited to them than mere 
frill. . . . The applicability of programming and machine 
teaching in problems beyond the level of simple mechanics 
remains an unexplored area. 

Approximately 60-65 percent of programmed mate- 
rial is on matlnmiatics and science; foreign language 
teaching comes next. Most material in English is on 
grammar, punctuation, spelling, vocabulary, reading 
comprehension, and such. In 1962, Doubleday pub- 
lished a Tutor Text on Modem Poetry, written by John 
Clark Pratt (Pratt, 1962), Instructor in the U.S. Air 
Force Academy, Colorado Springs. As far as I know 
very little has been done on programming literature. 
It is as though a vast new continent, just discovered, 
lies before you to explore. 

It is now possible for a machine or a programmed 
text to force students into the kind of close analysis 
of a piece of literature which English teachers have 
always wanted but rarely achieved. The obvious ad- 
vantage over the conventional classroom discussion 
method is that all students participate in all of the 
program. In a classroom usually a fc n’ carry the burden 
of the discussion. 

Try programming some small portion of what you 
teach. Your enthusiasm will grow as you proceed; 
you wiU be surprised and delighted with results; your 
teaching will never be quite the same again, for there 
will always remain with you that scientific, logical point 
of view in the presentation of your material. Here is a 
new aid to teaching, most promising when used weU. 
Try it! 
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TOP— TRYING OUT PROGRAMS 



Virginia Zachert 

Virginia Zachert outlines a five point evaluation effort for teachers who are interested 
in trying out programs. She states that programs must be helpful, meet teaching objec- 
tives, be practical and economical to use and produce statistical results similar to those 
being obtained from conventional instruction. She warns against over-sophistication 
in tryout design in a teaching situation. 



This field of programmed learning is something to 
preach about. I am completely convinced, after almost 
seven years of research in this and related areas, that 
programmed instruction offers something that can be 
helpful in all types of education whether religious or 
secular, whether formal or informal, whether academic 
or military, and whether adult or non-adult. You, too, 
are interested in new approach to education or you 
wouldn’t be here. 

In the field of education as in the field of psychology 
and reli^on it seems that everyone is an expert. So in 
the field of teaching machines and programmed learn- 
ing. Everyone is either “for ’em or again’ ’em” often 
with little or no knowledge of just what is being dis- 
cussed. Actually you can become an expert in this area 
with about 40 hours of reading because programmed 
iustruction is so new. However, the field is moving so 
fast that you are obsolete if you take time to sneeze! 

The most fascinating thing about this programmed 
instructional revolution is that now we have a way of 
not only checking on the student learning but of check- 
ing on our teaching. In many studies of educational 
techniques and the comparison of various methodology 
research has not shown results that were either desired 
or expected. The trouble with some of this is not in 
the experimental design but rather in the subject mat- 
ter. I was trained as an experimental psychologist and 
it is disturbing to have to work with humans. After all, 
people seem to be the most cantankerous things in the 
world, especially students! Although programmed learn- 
ing really holds great promise the “people” factor may 
be the reason research studies to-date haven’t shown 
dramatic results. 

Many people will tell you that the best thing for the 
teacher to do is “sit back and wait” until the researchers 
come out with more conclusive results and are able to 
tell just what is happening, how it is happening and 
what can be expected. But I say to you: Don’t sit back 
and wait! You are the onse who know what your needs 
are and you are the ones who can best tell what is 
answering these needs. So I say to you: TOP or TRY 
OUT PROGRAMS. 

There are five things that are going to be important 
as you enter into this area of experimenting or research- 
ing or evaluating. I will use the acrostic letters 
H O P E S to help you remember the five points. 



First, for the letter H, helps is your word. If pro- 
grams aren’t helps to you they should not be used. We 
look with high hopes to something that will help us 
to train better teachers. As you begin trying out these 
programs the first question will be: Does it prove help- 
ful? 

Secondly, for the letter O, I suggest that objectives 
be your word. Does this new method help you reach 
the teaching objectives you are striving for? Of course, 
here we may have to re-evaluate our objectives in our 
teaching and I think this might be good for us. What 
are our objectives in administering programs? Do we 
get objective data? How do we objectively evaluate it? 
So we ask: Does it prove objective? 

In the third place, for the letter P, I suggest that prac- 
tical be the word. In the field of research, of course, 
we usually can’t answer this question of practicality 
until the end of our work. As teachers you will be 
interested in it from the first day. Enou^ work has 
been done on programs that are now available for pur- 
chase that you can readily see if it will fit your needs. 
(After all, in programmed instruction, materials must 
be student tested as they are being prepared.) All of 
this information should be listed in “Programs ’62” and 
“Programs ’63” published by the Government Printing 
Office. So even though you may not know the answer 
at the start you definitely will want to ask the question: 
Will it prove to be practical? 

Fourthly, for the letter E, let the word be economical. 
One of the real problems in the field of education is that 
of money. With teaching machines this cost factor can 
be a real problem. The machines used at Keesler AFB, 
Mississippi, for three years to train electronic techni- 
cians were the U.S. Industries Mark I Auto Tutors. 
These cost a little matter of $5,000.00 each. We found 
these machines to be most valuable in controlling stu- 
dents, in keeping records of all choices, in recording 
time on various portions of the course, in storage capac- 
ity of over 10,000 pages per microfilm reel and finally 
in acting as a page turner. This last factor was not one 
of major importance for us. In fact, we had the mate- 
rial printed out in book form and administered at 
Keesler and found the books to be more expensive than 
the film. In programmed learning the machine in most 
school situations is not a problem of major importance 
at the present. For your experimenting I suggest that 
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books or even booklets would be much more sensible. 
But even these are expensive. If you start writing your 
own programs you will find that it is most expensive 
in the amount of your time that it takes. In fact, the 
cost of several hundred dollars a page has been quoted 
by some of the program writing outfits. Fortunately, 
most of the material you will be trying out will be in 
published form and will seU for little more than regular 
texts. But we must ask: Does it prove economically 
feasible? 

Finally, for the letter S, let me suggest statistics. Peo- 
ple, especially psychologist, love statistics. Many peo- 
ple, especially professional teachers, either fear or dread 
statistics. But one of the things you are going to find 
asked most often about any study you do is: “Was 
there a statistically significant difference between the 
two methods?” or “Just what did your data show”? 
My suggestion to you as practicians is not to worry 
about any elaborate statistical design for your try-outs. 
After all you are now able to tell whether your students 
are learning adequately or not. You are able to assign 
grades which are still found to be the best predictors 
of college or graduate academic work. And I am sure 
you can determine to your own satisfaction whether or 
not the methods you try out are valuable or not, and 
very simple statistical techniques are the tools you use. 
So don’t object to asking: Does it prove statistically 
valid? 

Here then are the five key words: Helpful, objective, 
practical, economical, and statistical which can be re- 
membered as we count them off on the fingers of our 
hand as we look with high hopes to the new field of 
programmed learning. 

Thus the thumb would be H for helpful, the index 
finger O for objective, the long finger P for practical, 
the ring finger E for economical and the little finger as 
S for statistical. Now look at your own hand and let’s 
count these off together raising each finger as we repeat 
the key words. 

This reminds me of a story. It seems that the fingers 
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were having an argument as to which was the most 
important. The index fingers says, “I’m most important. 
I’m used to point.” The second finger says, “Don’t be 
silly. Anyone can see that I am longer than any of the 
rest of you. Obviously you can see I’m most important. 
If a finger gets jammed its usually me.” The third fin- 
ger says, “No, I’m most important. I’m the ring finger 
that everyone looks at, especially on a good looking 
young girl”. The little piggy says, “Everyone knows 
I’m most important. I can get places that nobody else 
can get”. (At which time the little finger is inserted in 
the ear to clean it.) But the thumb says, “I’ve listened 
to aU you have had to say and I think you are each 
very valuable but I think we aU have strength as we 
work together and I consider this my most important 
function: to work with each of you”. (With this the 
hand is held up and the thumb is pressed against the 
tip of each finger.) 

So if the story is to have meaning to us perhaps I 
could say that all the key words are important. The 
first finger of objectives does point the way for us, the 
longer finger of practicality does stand out and often 
gets us in trouble, the ring finger of economics does 
draw attention to itself and certainly the little piggy of 
statistics can get facts that nothing else can uncover. 
But I think we would all agree that the thumb for bemg 
helpful is really the major key to our TRYING OUT 
PROGRAMS. We want helpful objectives, we must 
have helpful and practical material, we must help to 
keep it economical and within our budget, and the 
statistics must be helpful to our evaluation of what we 
have found out. 

Yes, this has been a TOP talk. And if you forget 
everything I have said don’t forget that only you can 
truly evaluate the value of programmed learning in 
your own teaching situation so as a professional re- 
searcher I challenge you as professional teachers to 
“come on in, the water’s fine”. You start trying out 
programs and give us the information we need for fur- 
ther research. 
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V. The Exceptional Student 

PROGRAMMED INSTRUCTION AND THE EXCEPTIONAL CHILD 

Willard Abraham 

Willard Abraham briefly summarizes current knowledge and misconceptions about 
the exceptional child. He points out several advantages of programmed instruction in 
meeting their specialized educational needs. However, there is a plentiful array of 
problems generated by this methodology. Some of the problems are generated by 
well-meaning people who try to sell programmed instruction as a panacea for the 
problems of the exceptional child. 



Until two years ago, I was more in my element talk- 
ing about the two extremes on the intellectual scale, 
the gifted and the mentally retarded, and their educa- 
tion. However, these two years have provided intensive 
experience in a relatively new interest of the prepara- 
tion of programmed materials, a subject with almost 
infinite potentialities for contributions to the broad 
area of education. 

One cannot be even remotely connected with the 
challenges of this field without recognizing some vital 
areas in which it will be felt with increasing intensity. 
The implications it has for all children are great, and 
those who give only lip service to the “all” factor and 
then promptly begin to make exceptions lose the point 
of its real potency. It can help do a better job of 
teaching than even our best teachers currently do, for 
it stresses the importance of the teacher as the core 
of the learning situation in this era of the knowledge 
explosion. 

It has a new, experimental, open-the-door, anti- 
status-quo lilt to it that in its vigorous way will ulti- 
mately bury the poor programs, inadequate and fakey 
testing, false claims, and sales pitches that temporarily 
threaten this basically strong, thriving infant. It has a 
glamor quality that attracts dilettantes and a revolu- 
tionary air that entices non-conformists from the fields 
of psychology and business — ^but it is important in its 
own right as a vital force in our schools, industries, 
military installations, and homes. It needs neither the 
rash and foolish statements that have tended to mar its 
auspicious birth nor the promoters who make them. 
Nor does it warrant the side-long glances of wonder 
prompted by a few of the slightly addled personalities 
that lend a little unnecessary color to a basically sound, 
thriving technique of teaching. 

Before we seek the implications of programmed in- 
struction for exceptional children, let’s stop for a few 
moments at the focal point of those children who devi- 
ate enough from the so-called normal to need some 
special educational assistance. As we discuss them you 
can be thinking (on the basis of what you know of the 



field of progr ammin g) of how programmed materials 
can help them perform closer to their capacities. 

HOW MANY OF THEM ARE THERE? 

WHO ARE THEY? 

WHAT DO WE KNOW ABOUT THEM? 

The answers here will have to be brief but direct, 
to-the-point, and indicative of the importance of the 
subject. 

We have given more than enough time to the cliches 
of education. It is refreshing that in one part of it 
words like the ones FU now use have meanings that 
are realistic — “individual differences”, “the whole 
child”, “the needs, interests, and abilities of children”, 
“take the child where he is”. The unfortunate point is 
that such words and phrases have sometimes been 
uttered almost indiscriminately, without real meaning, 
except in the area of special education where we at- 
tempt to talk in definite, meaningful terms about “chil- 
dren who differ”. 

The action needed is for a proportionately large 
number of children, approximately one in eight of our 
school-age population. If that number seems small to 
you, look at it this way: With the average size family 
in the United States consisting of three children, one 
out of your family plus the neighbors on either side 
of you will be “exceptional” — and don’t count on his 
being gifted! If that statistic doesn’t work out, there 
will be two exceptional children in the next group of 
three families and one will be in each group of eight 
or nine grandchildren stemming from each pair of 
grandparents, or two in the next group of grand- 
children. 

The mentally retarded, sight handicapped, speech 
handicapped, hearing handicapped, emotionally dis- 
turbed, orthopedically handicapped, and gifted, a group 
of frequently overlapping rather than discrete cate- 
gories, are among those under consideration. Their 
numbers and problems have received increasing atten- 
tion and understanding in recent years through women’s 
magazines, the concern of the late President Kennedy 
and his family, popular books (such as Pearl Buck’s 
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The Child Who Never Grew and Dale Evans Rogers’ 
Angel Unaware), motion pictures (such as A Light in 
the Piazza based on the New Yorker novella by Eliza- 
beth Spencer and A Child Is Waiting), and the dedi- 
cated professional people whose contributions bear little 
relationship to the limited financial returns they derive 
from the services they perform. But the neglect of these 
students continues in many localities and states. 

They represent tragic areas of false information and 
mis-mformation like these: (1) mentally retarded chil- 
dren are delivered to those who sin or to families in- 
variably tainted with at least a trace of retardation; 

(2) there is always a close relationship between in- 
tellectual limitations and the areas of mental illness, 
epilepsy, cerebral palsy, and brain injury; (3) “slow 
learners” always have low IQ’s. The truth is far re- 
moved from such statements: (1) mentally retarded 
children can be and are bom into families in every race, 
religion, nationality, background and socio-economic 
category; (2) there is no necessary relationship between 
mental retardation and the four groups listed above; 

(3) gifted children who perform below their capacities 
are literally “slow learners” of a type, and “average” 
children operating at a lower level are pseudo slow 
learners. 

Among the more neglected of the exceptional child 
categories (despite the increased conversation about it) 
is that of the gifted. With half of the top quarter of 
our high school graduates reportedly not finishing col- 
lege, with many teachers feelmg either uncomfortable 
or threatened by their classroom presence, with the 
foolish tag of “undemocratic” or “elite” stUl occasion- 
ally attached to special educational programs for them 
(although not for the football team, art class, band, or 
choir); the progress of our brigjht students still lags 
behind our need for their insights, potential knowledge 
and creativity. 

The 19th century feeling that they “ripen early and 
rot early” has long been laid to rest by the Terman 
study and others which indicate not only that as a 
group they are above average in all measurable traits 
but also that they generally retain their giftedness into 
adulthood. No longer can we be satisfied if they work 
up to grade level when they’re capable of more, or 
depend on their “muddling through” school when fig- 
ures show that too many drop along the educational 
wayside. No longer can we afford to be bogged down 
in definitions of who they are (IQ and/or talent and/or 
creativity), how to identify them, or what the termi- 
nology means (enrichment versus acceleration versus 
segregation, or heterogenious versus homogeneous 
grouping). No longer can we endure the talent wastage 
of bright girls (“she’ll only be a wife ... or mother”), 
of children from lower socio-economic groups, different 



cultures, or rural areas, of handicapped children who 
also happen to be bright and perhaps dijQficult to iden- 
tify as such because of their handicaps, of drop-outs 
whose finances, emotions, aspirations, or family con- 
flicts deprive them and us of their promise, of brilliance 
being rubbed down to mediocrity because of our 
neglect. 

Another problem area, already referred to briefly, is 
the “pseudo retarded”; the ones who are not really 
slow but who test out or appear to be at the retarda- 
tion level. The reasons are many and the solutions are 
not easy to develop. Among the causes are a different 
language or culture, financial depravation, invalid test- 
ing, physical handicaps, or (altihough we dislike ad- 
mitting it) poor teaching. Programmed instruction can 
certainly fit into this picture. 

The educational trends for these children and their 
implications for programmed instruction are fast- 
moving, vibrant currents on the school scene. For 
example, more than half of the fully blind children of 
school age are in regular pubhc school classes (with 
specialist “resource teachers” helping the regular 
teachers). The discussion of so-called sign language 
versus lip reading and speech for deaf children seems 
to be edging firmly toward the latter. Cerebral palsied 
children are now more frequently recognized as being 
part of the normal range of intelligence, although the 
average IQ for a large group of CP children chosen at 
random will be considerably below such figures. Termi- 
nology in the area of the mentally retarded has long 
ago left the moron-imbecile-idiot morass and settled in- 
to the more platable educable-trainable-custodial or 
mildly retarded-moderately retarded-severaUy retarded 
camps. 

Research in these areas is also making fantastic leaps 
of progress — ^with causes and correction for PKU (phel- 
ketonuria) and retrolental fibroplasia (blindness of pre- 
mature babies) as forerunners of other dramatic break- 
throughs. The awareness of and programs for the 
gifted, an increase in state aid to help support programs 
more adequately in all of these fields, and the coopera- 
tive, unselfish approach of professional disciplines to 
solve educational, medical, psychological, and related 
problems are all aflSrmative signs which point toward 
more progress in the years ahead. 

As we turn our attention to the programming side 
and seek implications and relationships between pro- 
grammed instruction and exceptional children, some 
vital points immediately come to our attention. 

In his remarks in his presidential address. Colonel 
Ofiesh put the issue in proper perspective when he said 
that through the use of this technique we “see children 
learn who otherwise would not have learned” and are 
able to “remove frustrating, devastating aspects of failure 
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to learn”. In a report at the NSPI Convention per- 
taining to mentally retarded children, Dr. F. C. Williams 
of the University of South Florida stated that hours 
could replace years in the area of word recognition; 
he also pointed out another unique development — 
materials prepared for use with the mentally retarded 
are being transferred to the education of the so-called 
normal, instead of the usual reverse pattern. Nowhere 
on the education scene is there more need for curricular 
materials based on small steps, sound reinforcement, 
and success than with the mentally retarded. No chil- 
dren have been so frustrated as they, so hurt, and ig- 
nored as they frequently are in the years before many 
of them get the special attention they need. Whether in 
regular classroom settings in isolated areas or in spe- 
cial education classrooms with skilled, well-prepared 
teachers; programmed materials prepared specifically 
for them and thoroughly tested on similar children can 
provide an educational breakthrough in this country. 

But there are problems. One is based on the tend- 
ency of many of us, and not just educators, by any 
means, to go off on national binges. We did it with 
prohibition, with the near-hysteria we sometimes dis- 
play toward movie stars, with our wild swings of the 
pendulum in clothing, food, and status symbols of all 
kinds. Martin Luther once said that humanity is like 
a drunken peasant who is always ready to fall from 
his horse on one side or the other. With cautious pro- 
fessional preparation, testing, and research in program- 
ming perhaps the danger of using progranuned mate- 
rials as a panacea can be lessened. 

Some of our more enthusiastic and less careful 
friends in this field can cause us the most difficulty. It’s 
the old line that your enemies won’t hurt you, you just 
need protection from your friends. A few years ago, 
Jacques Barzun wrote an article called “Cults of Re- 
search and Creativity” {Harper, October, 1960) in 
which he warned about mixing labels with real content. 
Our danger signs are related to his warnings as we tend 
to fall into a semantics mire. Too frequently some of 
our psychologists who are attempting to call the shots 
on programs and programming for school purposes are 
not school-oriented. Too often their lack of realism 
stems from the fact that their knowledge of children, 
child development, teachers, school practices and prob- 
lems, and administrative and financial issues is limited. 
Hiding behind terminology they arouse questions that 
are inappropriate and doubts that are unnecessary. 

Communication is for the purpose of sharing ideas 
and not for impressing people with one’s scholarliness. 
In both education and psychology we too often forget 
that simple objective as we deliberately mislead our 
publics with esoteric dogma. Some years ago Henry 
M. Wriston, former President of Brown University, 



demonstrated the obstructionist quality of language. 
He converted the song “Getting to Know You” (from 
The King and /) from its concise, monosyllabic clarity 
to this: “Motivated by empathy, Anna was restructur- 
ing empirically her approach to the technique of cross- 
cultural communication at a meaningful level with 
purposive intent. She undertook this in a relevant con- 
text of indigenous habitation.” 

An incident that occurred recently demonstrates a 
related issue. One of the top administrators in a large 
publishing firm, whose name you would recognize, said 
after a productive conversation regarding programs 
we might prepare for them, “I hope you don’t have 
any of those sandalled — ^bearded fellows working for 
you”. True or not, important or not, warranted or not, 
his inquiry is based on both truth and importance for 
him, and he constitutes a strong link in the program- 
ming future. 

Our objective whether we are talking about pro- 
grammed instruction for exceptional children, or other 
children, or industry, the military, or home improve- 
ment is to make a contribution, not create a cult, to be 
practical and not obscure, to help solve the plentiful 
problems we can identify related to learning and not 
to add to them. 

We face a plentiful array of other problems, such as: 

(a) teachers who are misled into thinking that they 
(or almost anyone) can write a program “that works”, 

(b) teachers encouraged to evaluate programs without 
proper guidlines for doing so, (c) limited field testing 
(as important as such evaluative procedures can be 
if done well) that is put forward as a substitute for the 
far more necessary controlled prepublication testing, 
(d) the lack of realization that a minimal level of apti- 
tude and ability are needed before a program can be 
used effectively, (e) the false assumption that a pro- 
gram developed for the so-called “typical” child will 
automatically work with exceptional children (“just use 
it with older mentally retarded or younger gifted or 
braille it or . . .”), and (f) the school problems of 
budget, fear of what’s new, apathy, or “what will I do 
with children who finish more quickly than others?” 

Programmed instruction can give real meaning to 
practices related to “acceleration” and “enrichment” 
often glibly referred to among gifted child educational 
plans. It can increase individual differences as it 
encourages children to move at their own rates and into 
new areas of knowledge and wisdom. It can help re- 
store the battered-down confidence and help reduce 
the accumulated frustrations of our “slow learners” 
and retarded. 

“A man’s reach should exceed his grasp, or what’s 
a Heaven for?”, said Robert Browning, and the advice 



89 



Trends In Programmed Insiruciion 

is sound for those of us who fully appreciate the con- 
tribution we can make in a very concrete way to spe- 
cific children. “A teacher can never better distinguish 
himself in his work than by encouraging a clever 
pupil”, said Carolus Linnaeus, “for the true discover- 
ers are among them, -as comets amongst the stars”, 
and we begin to recognize that this new approach to 
learning fits right into that teacher’s hands. From an 
unusual source comes the final bit of encouragement. 
In a book introduction, Danny Kaye could have added 
“education” to the sequence when he wrote that “you 
can’t bring health and happiness to a million children 
by signing a paper or waving a wand. It has to be 
done child by child”. He might very well have been 



talking about programmed instruction and exceptional 
children, as the technique and recipient of the child-by- 
child route to educational accomplishment. 

There is a shortage of sound programs and espe- 
cially for the exceptional. There is currently only a 
limited number of teachers who know how to use pro- 
grams wisely and especially with the exceptional. 
TTiere are still few school administrators who recog- 
nize the full value of programmed materials and es- 
pecially for the exceptional. All of the problems related 
to production, testing, distribution, public relations, 
and acceptance apply in intensified form to the children 
who still remain on the fringes of our educational con- 
cern in many parts of the country. 



THE DRAPER EXPERIMENT: A PROGRAMMED LEARNING PROJECT 

John M. McKee 

John McKee describes an experimental project in education and rehabilitation at 
Draper Correctional Center, a prison for youthful offenders in Alabama. The new 
educational technology of programmed learning is employed with 125 inmates in a 
Self-instructional School. The primary aims of the project are: (1) to develop a low cost 
self-instructional program that will incorporate into its operation all educational, voca- 
tional and rehabilitation training; (2) to design, develop, and utilize a seif-instructional 
curriculum more applicable to the educational, occupational, and social needs of men 
in a prison society, both with respect to that society and the one they will be preparing 
to enter outside; (3) to evaluate the program both with respect to performance in 
prison and following release; and (4) to prepare a useful and adequate “blueprint" 
that can be adopted by correctional and other institutions. 



This is a story abon; a dramatic application of Pro- 
grammed Instruction to a special area — a prison for 
youthful offenders at the Draper Correctional Center 
of Elmore, Alabama. To my knowledge, we operate 
the only full-time self-instructional school in the nation. 

The Draper Experiment was proposed to the Ala- 
bama Board of Corrections as a reasonable approach to 
the rehabilitation of inmates by Watkins and McKee 
(1960). The project was approved and it was initiated 
on 15 March 1961. The purposes of this project in 
education and rehabilitation were (1) to demonstrate 
the administrative feasibility of an entirely self-instruc- 
tional education program within Draper — evidenced by 
an ongoing project, the ease with which we could 
interest and teach the inmate students, and the degree 
to which institutional resources could be used; and 
(2) to make a tentative analysis of the subject matters 
and skills most practical to teach inmates from the 
standpoint of obtaining the most productivity at the 
least cost. 

It should be made clear that at the outset we related 
educational and vocational achievement of the Draper 
inmate to his rehabilitation, both inside and out of 
prison. We made the assumption that positive and 
significant educational and tr ainin g accomplishment is 
a factor important to the modification of his behavior 
and attitudes. The institutionalized “first-offender" 
typically comes from a lower class background, the 
pressures of which evidence themselves with an in- 
ability to tolerate the actual role of student. This is 
sufficient in many cases to retard or eliminate desired 
educational development; nevertheless, intellectual and 
educational development is a need expressed by most 
inmates. There is, then, a discrepancy between ex- 
pressed wishes for educational status and actual 
achievement in the middle class learning situations 
foimd both inside and outside the prisons. Thus, the 
inmate, although isolated from the main stream of so- 
ciety, agrees, in principle, with the value placed upon 
education and training as a fundamental requirement 



for successful living. But the inmate lacks the psycho- 
logical equipment to instrument his motive. Further- 
more, a good case can be made for relating the process 
of academic education and job preparation training to 
recidivism in the young offender and for demonstrating 
how new techniques of education hold promise for 
contributing to the goal of recidivism reduction. 

The project was begun with 25 inmate volimteers 
approved by the Warden. The Warden’s principal cri- 
terion of selection was that subjects could achieve, in 
his judgment, at a high school level. A significant num- 
ber of “troublemakers” were purposely chosen for the 
group in order to assess any potential effect on this 
type of prisoner within the institution. 

The demonstration project had been proposed by the 
Division of Mental Hygiene of the Alabama State 
Health Department, to the Alabama Prison System. 
The major cost of the first three months of operation 
was borne by the Alabama Prison System, which con- 
tracted with Educational Design of Alabama to assist 
with this phase of the project. During its first year of 
operation the project received continuing support from 
the Prison System, the Division of Mental Hygiene and 
from two grants from private foundations. 

From the pilot project that ran for a year, McKee 
and Slack (1961) made a number of interesting dis- 
coveries: 

1. Self-instructional programs can be successfully 
used with confined yoimg offenders. The success of 
self-instructional programs with delinquents is remark- 
able, considering the almost complete lack of success 
of other types of material with this population. Yet, 
from a strictly behavior theory standpoint, it is not diffi- 
cult to understand. Individuals who have a history of 
repeated failure in normal classrooms have been con- 
tinually deprived of that certain reinforcer which we 
call “success." They are rarely if ever right in the 
answers they give on tests to teachers’ questions, home- 
work assignments, and quizzes. In some cases this 
reaches the point of extinction on all formal learning 
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tasks and even on attendance at such tasks, resulting in 
truancy and drop-outs. Here is a population of men, 
with an average age of 22, who have repeatedly failed 
in whatever they undertake. They failed in their homes 
(or the homes failed them!), in their neighborhoods, 
in school, on jobs; in fact, they have even failed in 
crime, or why else would they be in prison! While 
going through a self-instructional program, on the other 
hand, the average student finds ifiis answers correct 
more than 90 per cent of the time. Such success is 
indeed enough to maintain behavior relevant to pro- 
gressing through the materials for periods of time longer 
than would be anticipated with “brighter” and more 
successful students. 

2. Another finding of the pilot project was that the 
experimenter-subject role relationship, wherein the sub- 
ject is a cooperative research partner in a new venture 
rather than a more or less passive student, appears to 
be the best role-vehicle for maintaining task-relevant 
behavior. The Draper Project has been largely inmate 
administered, inmate managed, and inmate supervised. 
Such a system which gives the inmate maximum re- 
sponsibility, status, and self-determination appears to 
be markedly superior to alternative conventional stu- 
dent roles which contain previous negative associations. 
I'he Draper inmates seem enthusiastic over the prospect 
of serving as subjects in a unique experiment in which 
they are fully involved and are the chief determiners 
of the outcome. 

3. A very interesting phenomenon developed early 
in the experimental group; the intellectual interests of 
the subjects expanded very rapidly, and they were 
maintained at a high level of intensity. Most of these 
interests were noted by the experimenters and were 
reinforced as quickly as possible by obtaining books 
for them to read and by securing certain resource mate- 
rials which were not readily available, such as diction- 
aries and technical books. The warden has stated on 
several occasions that this dramatic mushrooming of 
interests stands in marked contrast to attitudes of the 
inmates prior to the initiation of the project. For 
example, it was also noted that large numbers of the 
inmates expressed a keen interest in obtaining a college 
degree, whereas earlier many frankly admitted that they 
had not the slightest interest in pursuing an advanced 
education. 

4. It is very likely Aat prisoners can produce self- 
instructional programs themselves. After a few weeks 
of taking the courses, several inmates started writing 
them. Some of the inmates are highly intelligent indi- 
viduals and have attained a great deal of skill in pro- 
gram writing. Besides, they seem to have the perse- 
verance that is essential for this task; and, as they state 
it, they have “plenty of time” in which to do it. 



5. Inmate interest in serving in the Armed Forces 
could be an important factor in the rehabilitation chain, 
provided these men become more generally accepted 
by Selective Service. Skty-seven per cent of the sub- 
jects expressed a desire to serve in the Armed Forces 
or clear up a bad record that was made in military 
service. 

6. There are a great number of significant reinforc- 
ing factors operating in such a project, and it would 
seem desirable for the professional staff and the 
warden to be in full control of them. A very important 
factor is the type of feedback the subject receives as he 
progresses through a course. Very few programs have 
accompanying tests, so inmates can only guess at their 
progress. Some of the experimental subjects are devel- 
oping tests themselves for the various courses. Cer- 
tainly, pre- and post-test procedures should be in- 
duced as quickly as possible — not only for better 
feedback to the inmate, but also for the evaluation of 
the project itself. 

In May, 1962, the Draper Project received a three- 
year grant from the National Institute of Mental Health 
to conduct a more extensive education and rehabilita- 
tion program. 

AIMS OF PRESENT PHASE OF PROJECT. Un- 
like other research investigations into the use of new 
techniques in education and rehabilitation, the primary 
aim of this project is to develop a set of procedures 
for establishing and operating an administratively feasi- 
ble self-instructional school in a correctional center. 
Thus, a major result of the project will be a “blueprint” 
report series for use by institutions who may wish to 
take advantage of the modem technology at low cost. 
Sub-goals are: (1) to design a self-instructional cur- 
riculum more applicable to the educational, occupa- 
tional, and social needs of men in a prison society, both 
with respect to that society and the one they are pre- 
paring to enter outside; (2) to integrate useful social 
experiences with the specific training; (3) to develop 
a systematic and extensive coordination of such a proj- 
ect with local school boards and with other local and 
state agencies, to capitalize upon these relationships 
by investigating more thoroughly, experimenting with 
ways of developing positive attitudes toward self- 
instmction and generalizing that attitude to other more 
traditional rehabilitation procedures; (4) to study both 
intrinsic and extrinsic reinforcement factors to deter- 
mine how best to maintain inmate motivation at an 
optimal level; and (5) to evaluate the effectiveness of 
the program by following up released offenders. 

The inmate population has so far been highly moti- 
vated by the fact that their performance in the self- 
instmctional programs is needed in the evaluation of 
these programs; and they are to some degree pioneers 
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in a new technological development which is having 
great impact on public education, industry and the 
Armed Forces. Both student and staff have come to 
realize that this population is for many purposes an 
ideal testing ground for new programs as they are 
developed — not in the sense that it is representative, 
but in the sense that it presents in a controllable situa- 
tion a wide range of educational rehabilitation prob- 
lems. 

Self-instructional techniques and other recent devel- 
opments in educational technology have some specific 
features which are unique and valuable. For example: 
(1) the immediate knowledge of results obtained from 
all self-instructional programs appeals to the need for 
immediate gratification in the inmate population, as 
does the high rate of success inherent in linear or 
Skinner-type programs, Crowder’s intrinsic programs, 
and Gilbert’s mathematical lessons; (2) the minimum 
use of teachers and the role change from student to 
experimental subject cuts through the existing class 
educational barrier and provides for “Hawthorne 
Effect” benefits; (3) machines have been shown to 
have a motivating effect on populations (military) 
which are not dissimilar to the adolescent inmate popu- 
lation both in regard to individual and group factors; 
(4) lack of competition, absence of embarrassing dis- 
closure of ignorance, and the self-pacing features of 
self-instructional techniques have been demonstrated to 
be decided advantages with apparently unmotivated or 
recalcitrant individuals; and (5) another important in- 
centive from the inmate point of view is the fact that 
during confinement he can make up lost educational 
attainment, in many cases learning exotic subject matter 
(e.g., the German language) not generally achieved by 
those with less time to spend. 

SIGNIFICANCE. The adult delinquent is largely 
cut off from normal routes to achievement, and one 
very important factor in his virtual unemployability in 
our technological society. Through a specific course of 
trade and academic education that can be practically 
individualized to his specific needs, largely because of 
the economy of the self-instructional method; we be- 
lieve that we can secure these accomplishments: (1) 
put into the young adult delinquent’s hands specific 
instruments for occupational and school achievement 
he will need outside the institution; (2) add appreci- 
ably to his self-enhancement, insofar as such develops 
out of the experience of achievement; (3) add appre- 
ciably to his motivation for more normal social adjust- 
ment, insofar as the behaviors of such an adjustment 
will come to appear to him as valid instruments for 
attaining normal goals; and (4) supply him with more 
intrinsically interesting self-development activities; per- 
mitting us to arrive at a greatly improved prescription 



for creating a positive institutional attitude on the part 
of the inmate and thus relieving the prison of a heavy 
administrative burden. Our previous work in this area 
gives us confidence that these things can be accom- 
plished. 

Recognizing that a well-planned program of dis- 
semination of results is important in an activity of this 
nature, we plan to: (1) make available information on 
the administration, cost, and expected results of such 
a program in a prison to public schools, mental hos- 
pitals, TB sanitoria, etc., who might find ancillary value 
in it; (2) make available guidelines and ways to obtain 
the cooperation of public schools and colleges in an 
educational venture of this sort; (3) make available in- 
structional programs developed at Draper; and (4) act 
as a testing group or “proving ground” for newly de- 
veloped programs. 

RECENT PROGRESS. Although the Draper Experi- 
mental Project in Education and Rehabilitation has 
been under way since 15 March 1961, it was not on 
a full-time professionally staffed basis until 1 May 
1962, when a grant from the NIMH was received. This 
made possible the full-time appointment of both the 
Project Director, Dr. John M. McKee, formerly Direc- 
tor of the Mental Hygiene Division of the Alabama 
State Department of Health, and the Vocational Spe- 
cialist (electronics), Norman Ussery. 

The flow of students into the Self-Instructional 
School (as it is now called) rose rapidly and soon was 
more than doubled. In addition to the self-instructional 
group, for which the courses are primarily at high 
school level, the elementary classes are now located in 
the new school area. This new group, which has in- 
mate instructors, is called the Junior Division of the 
newly created Self-Instructional School, and the com- 
pletely self-instructicnal group has been named the 
Senior Division. 

One of the first moves to upgrade and improve the 
Junior Division was to introduce ail the programmed 
courses available. It is anticipated that within a few 
months many more of the courses wiU be taught on a 
self-instructional basis. However, the teaching of illit- 
erates to read and write is a problem to which pro- 
grammed learning has not yet made any significant 
contributions. The Laubach method, we discovered, is 
a very effective instrument for literacy education of 
prisoners. A corps of inmate instructors has been 
trained in the use of the Laubach method. These in- 
structors are teaching fellow inmates with excellent 
results. 

With the large increase in student body and the 
availability of many new programmed courses, it was 
immediately apparent that more space was required. 
By December 1962, an additional adjacent area was 
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obtamed and converted into classrooms. One of the 
new rooms was made into a large language arts labora- 
tory, where the students in the Senior Division learn 
Englis^i, reading, speed reading, foreign languages, and 
vocabulary development. Each student is required to 
spend at least two hours a day in the laboratory. This 
phase of the School’s program is managed by two in- 
mates assigned one-half day each. A total of 70 in- 
mates flow daily through this lab. 

Although programmed materials prepared for self- 
instructional use are becoming available on a wider 
variety of subjects, as well as at different grade levels, it 
was found when evaluation of these programs was 
undertaken, that most of them had no pre- or post-test 
examinations. Thus, it was necessary to develop such 
instrument. Now there are no programs in the school 
for which we do not have examinations. 

During the past year the Draper Project has received 
national recognition, and this has brought a large in- 
crease in the number of visitors from all over the nation, 
as well as from Alabama. 

The project is committed to programming the vari- 
ous vocationcil trade areas taught at Draper. The field 
of electricity and electronics was chosen as the first to 
be programmed. To date the following areas have been 
developed: 

Course Title Number of Frames 



Basic Electricity 



Introduction 


500 


D.C. & Ohm’s Law 


600 


Meters 


500 


Batteries 


490 


Magnetism 


750 


Alt. Current 


800 


Motors & Generators 


650 


Basic Electronics 


Resistors 


400 


Transistors 


800 


Mathematics for Technicians 


Automotive 


500 


Carpenters 


500 


Electricians 


800 


Radio & TV Repair 


Self-instructional 
workbooks (4) 


Residential, Commericial 


Self-instructional 


& Industrial Wiring 


Manuals 



These courses were programmed in their entirety at 
Draper, utilizing a corps of inmate programmers trained 
in programming techniques and supervised by Norman 
Ussery, the Project’s specialist in electronics. After a 
man completes his basic electricity training — both 



theory and lab work — ^he is assigned to an electrician’s 
squad. As a member of this squad he has assignments 
in wiring, installing electrical circuits, and trouble 
shooting throughout the prison. Before release the stu- 
dent is aided in getting a job as an electrician’s helper 
in the city to which he is released. To date, two of the 
graduates of this program have been released and are 
now employed as electrician’s helpers. 

Another vocational area in which programming is 
being done is mechanical drafting. A mechanical draw- 
ing and blueprint reading course is being taught by an 
inmate expert in this field. This subject is provided 
primarily for the electricians who must have this skill 
as part of their training. When completed, this course 
will be self-instructional but will require a laboratory 
supervisor. Finally, a short course called “Signed Num- 
bers” has been completed. 

Since the project is almost entirely dependent upon 
the inmate for all types of services that the School re- 
quires, whether instructional, clerical, or progra mm ing; 
it has been necessary to develop a corps of inmates to 
perform these functions. Thus came into being the 
School’s “Service Corps” comprised of 25 men. Each 
member of the Corps, in addition to performing his 
service functions for one-half day, is required to spend 
the remainder of his time in the School as a student. 
The responsibilities of the Corps members are: 



Type of Service Number Assigned 



Clerk 3 

Instructor 5 

Programmer 6 

Subject-matter Counselor 8 

Librarian 2 

Lab Technicians 4 



One of the key and unique Service Corps members 
is the subject-matter counselor. This person serves in 
one of four areas: math, language arts, science, or elec- 
tricity. Usually, two men are assigned to each area. 
The language arts counselors, for example, manage the 
language lab and direct or assist all the students that 
flow through that room during the day. They keep 
progress charts on the student, determine when he is 
ready to take a test in a particular course and put to 
use the curriculum of the labs. The subject-matter 
counselors are frequently called upon to construct tests 
in the various courses of their specialty area. Finally, 
the student must obtain the counselor’s approval before 
he is allowed to take a final test. The counselor must 
be confident of the student’s making the passing grade 
of 85 per cent, for the counselor is held responsible 
for the student’s failure. 
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The counselor is developed into a subject-matter 
expert before he is selected or placed in this capacity. 
The mathematics counselor, for example, must have 
completed the entire high school mathematics curricu- 
lum and a number of advanced courses before he is 
considered an expert and given his job. He must show 
unusual interest and ability in mathematics, and be well 
liked and respected by the other inmates. Further, he 
must be willing to give half of his time to helping 
others. Similar requirements are demanded of coun- 
selors in other areas. 

The subject-matter expert is also the chief test con- 
structor. Since many of the courses that are commer- 
cially available do not have tests made up for students, 
and because students need and demand testing feed- 
back, it has been necessary to develop over 100 tests. 
All tests are reviewed and edited by the project staff 
before they are approved for classroom use. Since 
some of the courses — ^particularly electricity, algebra, 
and plane geometry — are very long, it was discovered 
early that the student and counselor need more frequent 
feedback to be assured that the student does not have 
blind spots and to be assured of passing the final test 
with 85 per cent correct response. Therefore, the 
longer courses now may have 15 or 20 subtests (meas- 
uring acquirement rather than accomplishment y the 
latter being the function of the final exam). After the 
student takes a subtest the counselor reviews all mis- 
takes with the student and teaches him the material he 
must learn to attain perfection; or the counselor may 
direct the student back over a certain part of the course 
that he did not master well. After the student has 
satisfied the counselor, he is given permission to con- 
tinue through the next group of frames. 

Not all test construction men are counselors, how- 
ever. A few exceptional and advanced students are 
asked to construct tests, which they very willing do 
because of the prestige associated with the task and 
their desire to improve the School. 

The Service Corps, to be sure, commands the highest 
status in the School. Its membership is chosen on the 
basis of competence, and all know this and respect the 
Corps for it. The counselor and instructor, for ex- 
ample, discuss and evaluate frequently the students 
under their direction with a “free-man” — something 
rarely done in prisons. Ordinarily, this would be called 
“putting a man down” or “ratting” But not so in the 
School. The reason for its acceptance, far from being 
mysterious (though visitors from other prisons continue 
to be amazed!), rests in the one central demand placed 
on the student — or counselor for that matter — namely, 
that he develop competence. Nothing else will do. Any 
evaluation made of him, from learning problems to be- 
havior problems, is objectively and rationally ap- 



proached. It is not a matter of moral training per se 
(though education in responsibility may be taking 
place), nor of “good” behavior, nor of general con- 
formity from fear of authority. Only productivity and 
development will suffice! 

Managing the intellectual productivity of over 125 
prisoners by only two “free-men” from 7:(X) A.M. to 
4:00 P.M., is a complicated task indeed; however, the 
procedures that have been developed for control of 
productivity, though elaborate, are entirely functional 
and yield results of a gratifying sort. 

First of all, no difficulty is experienced in recruiting 
students for two principal reasons: (1) assignment to 
the School carries prestige and students like it; and 
(2) many of the other assignments at Draper require 
strenuous labor and are both monotonous and un- 
pleasant, especially farm work. So, all students are 
volunteers. 

But once the new student enters School there is no 
assurance he will work consistently. The novelty of 
programmed learning has long since worn off at 
Draper, though we remain certain this format of sub- 
ject-matter presentation is far superior for our pur- 
poses. Even though we are convinced that programmed 
material, with its so-called intrinsic reinforcing quality, 
has definite limitations in motivating students (for ex- 
ample, through five volumes of TEMAC Algebra), we 
can say that the programmed format still holds interest 
in the student who has completed literally thousands 
and thousands of frames. In fact, the advanced student 
in self-instruction will rarely study a subject in conven- 
tional texts. He reports that doing so “wastes time” 
or is “dull”, particularly when a programmed text in 
the subject is available. 

When the new student enters school, he undergoes 
an orientation. He is told by a member of the Service 
Corps, one of the School’s orientors, what he can ex- 
pect and what is expected of him. (This orientation 
procedure is now being recorded and will soon be pro- 
grammed.) Next, he is interviewed by the Project Di- 
rector or his assistant, who determines the exact evalua- 
tion tests that should be administered to the new 
enrollee. All entering are required to take both an aca- 
demic achievement test and a vocational interest test. 
From interview and objective data an individual “diag- 
nosis” is arrived at. Then a “prescription” is prepared, 
which details his initial course of study. This will in- 
clude in all cases training in language arts and math, 
but a student’s overall personal-social rehabilitation 
needs are considered and other training steps are 
planned. 

For the Senior Division, a typical assignment on the 
new student’s Course Data Sheet is as follows: Reading 
(SRA), Spelling (Sterling), English 2600, Vocabulary 
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Development (Coronet, SRA), 7th Grade Math 
(TEMAC). The student requires constant evaluation 
and review of his progress, rather than direct super- 
vision. This is accomplished by the counselor’s fre- 
quent contacts with him and with the Project Director 
or his assistant. The student himself is given a “prog- 
ress plotter” for each course. This plotter has a “par 
angje” that he must stay within — this angle having been 
determined empirically for each course. If the student 
falls below minimum slope, the counselor moves in to 
discover why. Perhaps the rate is slow because of a 
reading difficulty. If so, special concentration is made 
in this area. If he has momentarily slacked off in his 
work, the counselor spends more time with him; and 
the professional staff will give him more attention which 
is sometimes all that is needed in the first place. 

Rehabilitation Efforts. Rehabilitation is a term with 
many different meanings, but in the field of corrections 
it can only refer to a significant reduction in recidivism. 
No other meaning carries as much weight with the pub- 
lic which must foot the bill for the law enforcement 
agencies, courts, and prisons. 

So it is that all factors relevant to a prison’s rehabili- 
tation program are viewed from the standpoint of their 
contribution to recidivism reduction. Some important 
factors include: custody, physical plant of prison, busi- 
ness management of the institution, prison personnel, 
specific correctional education, as well as vocational 
and general education. 

This project, from its inception, has accepted the 
principle that it is a part of and deals with the total 
prison complex. The project’s chief contribution to 
date has been a unique approach to general education 
and vocational training. The enormous implications are, 
of course, obvious to a man who has faued in every- 
thing — even in crime! The very rapid development of 
useful skills and knowledge, which the project in this 
short time has already effectively demonstrated, has 
radically changed many prisoners. The program holds 
significance for ego enhancement, as well as develop- 
ment of an appreciation for intellectual competence, 
both of these being a deeply rewarding success experi- 
ence so long denied the typical prisoner. And this 
whole change can now be engineered and directed 
surely and quickly by self-instruction. For example, a 
total of 22 self-instructional students have retaken the 
California Achievement Test and have registered a gain 
of three grades. The total time of association in the 
program was only 5.5 months. Should this increase 
hold up for others, the project will have demonstrated 
the effectiveness of programmed learning for at least 
very significant advances in grade level achievement 
over amazingly short periods of time. 



At the same time, this project is committed to educa- 
tion that has specific implications for “correction,” or 
desirable behavioral change. It is a reasonable assump- 
tion that the following skills are very relevant to the 
repertoire of a rehabilitated person: personal grooming 
and etiquette, personal responsibility for one’s behavior, 
good posture, correct spoken English, more socially 
acceptable attitudes towards the opposite sex, manag- 
ing a personal or family budget, and holding a success- 
ful job interview. We refer to these training areas as 
“adjunctive behavior skills”. They are adjunctive to the 
two major commitments of the project — ^general educa- 
tion and vocational training; but in another sense they 
may be critical to the rehabilitation picture. Take for 
example the matter of good speech: It is reasonable to 
hold that the ex-prisoner who desires to “go straight” 
will be immensely aided by having his subject and verb 
agree, avoiding slang, clearly enunciating syllables, and 
developing oral automatisms in correct speech. 

Realizing the high value of this particular adjunctive 
behavior skill, the Project Director initiated an experi- 
mental class in spoken English which meets one hour 
daily. This class has six inmates serving as student 
subjects. The purpose of the study is to discover the 
content for a course designed to modify incorrect speech 
and the techniques to bring about this modification 
quickly. It is anticipated that a programmed course will 
soon be developed from the experiences of this pilot 
class. 

Future Plans. Nov/ that many of the mechanics of 
establishing and operating a self-instruction* i school 
have been worked out, the project will address itself 
more to specific rehabilitation areas, such as social 
skills, attitude changes, etc. Along this line, plans are 
being made to accomplish the following during the next 
twelve months: 

1. Program a basic course in spoken English that 
will significantly modify the speech habits of the youth- 
ful offender in a relatively short period of time; 

2. Program the eight parts of speech; 

3. Put in final form self-instructional courses devel- 
oped at Draper in ‘Tersonal Grooming and Etiquette” 
and “Rules and Regulations of Parole”; 

4. Develop other needed courses, such as interview- 
ing for a job, the “Convict Culture”, a programmed 
orientation to Draper, and a course in “How to Stay 
Out of Trouble”. 

Great strides have been made since May 1962 to im- 
prove the curriculum of the School. During the next 
twelve months, a four-step program will be taken to 
boost further the course content of the School: 

1. A more balanced education will be developed by 
the introduction of new courses in literature and history. 
A science lab will be constructed to house already 
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purchased equipment and courses now taught in various 
science fields. 

2. There will be an increased use of the Laubach 
method of literacy training, and explorations into the 
other approaches to the problem of illiteracy. 

3. A survey study of the literacy problem at Draper 
will be conducted by two graduate psychology students 
at Auburn University, who will incorporate their find- 
ings in Master’s theses. 

4. Programming in the general field of electricity 
and electronics will continue until the entire trade area 
is completed. A basic course in mechanical and archi- 
tectural drafting for electricians will be programmed. 

We have related educational and vocational achieve- 
ment of the typical inmate as a necessary adjunct to 
his rehabilitation, both inside and out of prison. We 
have described the prisoner as having a history of re- 
peated failures in life, observing that he functions gen- 



erally on a success deprivation schedule. We propose 
to intercede his life space with ‘significant success ex- 
periences in critical areas — academic, vocational, and 
social. Programmed learning is the principle for this 
skill training. 

We have suggested that an adequate rehabilitation 
program (the end product of which is a significant re- 
duction in recidivism) requires not only specific train- 
ing in the above areas, but also a vigorous, intelligently 
conceived, and adequately funded follow-up program. 
An inmate who has participated in our program for a 
number of months will possess a certain amount of 
momentum to succeed in free society. He will have 
optimism and drive that should carry him along over 
the initial adjustment period. This outlook should be 
reinforced by concerted community planning and effort. 
As our project moves along, this area must be sys- 
tematically approached. 



PROGRAMMED AUDIO-VISUAL TRAINING FOR THE MENTALLY RETARDED 



Edmund C Neuhaus 



Edmund Neuhaus describes a successful project of programmed audio-visual training 
for mentally retarded young adults in the highly competitive electronics industry. This 
project is unique in that it teaches motor skills and seeks to make them productive 
self-respecting contributors to the community. 



A unique research and demonstration project, sup- 
ported in part by the Vocational Rehabilitation Admin- 
istration, employing mentally retarded adults is being 
carried out at Abilities Inc. The major goal of the 
project is to determine the feasibility of employing 
mentally retarded adults in a normal competitive indus- 
trial setting. One of the primary concerns of the 
project’s methodology is exploring the effectiveness and 
possibilities of teaching the retarded worker vocational 
skills by programmed audio-visual training methods. 
This paper will describe the preliminary experiences 
and developments regarding the use of programmed 
audio-visual methods in training ine retarded worker. 

The setting of this research study is Abilities Inc. of 
Long Island, New York. Under the dynamic leadership 
of its president and founder, Henry Viscardi Jr., Abil- 
ities Inc. employs 480 severely disabled persons on 
electronic assembly contracts. Competing successfully 
with local industry, this electronics plant is now training 
and employing mentally retarded young adults on in- 
dustrial and commercial job operations. From the 
Human Resources Fundation, the research and teaching 
division of Abilities Inc., the necessary professional and 
supervisory staff is obtained. 

The population comprising the research group in- 
cludes 25 mentally retarded individuals between the 
ages of 18 and 30, with mental levels that fall between 
the I.Q. ranges of 60 and 80 as obtained by the Wech- 
sler Adult Intelligence Scale. Their academic skills are 
at the 3rd to 5th grade level of achievement in basic 
subjects. They display neither emotional or organic 
problems that would interfere with a work adjustment. 
The entire group is well motivated to work and be pro- 
ductive in a normal competitive setting. In general, the 
intellectual functioning of these retardates is in the 
educable or high level range of mental retardation. 

The retarded worker is placed in a department of 
Abilities Inc. that will best suit his skills. Each worker 
is trained on at least four or five different job opera- 
tions. The retardate works alongside the normal 
worker. In no way is he isolated and protected from 
the competitive demands of industry. A training super- 
visor, who is a professional in the field of retardation 
and knowledgeable in industrial skills, maintains a 
close relationship with the foreman and other indus- 
trial supervisory personnel so that the retarded worker’s 



intellectual functioning and background is fully under- 
stood by them. The training supervisor initiates the 
conventional industrial training period with the retarded 
worker, and continues the training relationship until 
the retarded employee demonstrates an acceptable de- 
gree of efficiency. At that time, the retardate becomes 
a part of the regular productive work force. Close con- 
tact is constantly maintained between the training super- 
visor and industrial personnel. 

The retarded employees are presently working on 
job operations that include soldering, electrical and 
mechanical assemblies, wire stripping, tinning, flagging, 
ferulling, and laying and lacing of harness and cable 
assemblies. On one job contract, the retarded are per- 
forming skills th?* ' — lude stacking of laminations, 
packing and lacing of rotors and stators, assembling 
brush riggings, impregnating the rotors and stators, and 
then completing the assembly of this specific generator. 
The retardates are working on similar electronic assem- 
bly skills on job contracts from companies including 
IBM, ‘Republic Aviation, McDonnell Aircraft, Bosch 
Arma, Ford Instrument, Western Electric and Reming- 
ton Rand. 

The project is now exploring the use of programmed 
audio-visual training methods in teaching the retarded 
those job skills found in an electronic assembly plant. 
The rationale for using these techniques lies in the fact 
that local electronic assembly plants on Long Island, 
such as Sperry, Republic Aircraft and others, reported 
a significant decrease in training time and an increase 
in production quality for normal workers when employ- 
ing these techniques. Also, the general success that 
audio-visual training has enjoyed in academic circles 
encouraged its experimentation with the retarded. Pro- 
grammed audio-visual approaches have been used 
minimally in teaching academic subjects to the retarded. 
In a recent survey of the available literature, Stolurow 
(1963) found only 14 research projects using pro- 
grammed instruction for the mentally retarded. These 
projects dealt primarily with reading, spelling, lan- 
guage, arts, arithmetic and reasoning. Their use in 
teaching vocational or motor skills to the retarded adult 
is clearly a pioneer and uncharted area. Even in the 
teaching of motor skills, per se, to normal adults, audio- 
visual methods have been used only to a very limited 
degree. 
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The programmed audio-visual phase of this research 
study has proceeded cautiously and slowly. The equip- 
ment chosen was the Videosonic audio-visual aids pro- 
duced by Hughes Aircraft. This is an easily operated 
machine that accommodates slides placed in an auto- 
matic feeding cartridge and gives simultaneous audio- 
type instructions. The slide presentation can be paced 
by the individual’s own needs. An audio-visual train- 
ing area was set up with four Videosonic machines and 
individual work stations that are completely equipped 
with appropriate work materials. 

The first exposure given the retarded population with 
audio-visual equipment consisted of an orientation or 
trial run assembly of a “rabbit ear” television antennae. 
This trial run afforded both die retarded group and the 
investigators an opportunity to become familiar with 
the “hardware” under actual operating conditions. 
Moreover, it allowed the investigators to obtain an 
initial reaction from the retardates concerning this new 
and novel training approach. 

The television antennae assembly program included 
35 colored slides with audio-tape instruction. This pro- 
gram was developed by Hughes Aircraft Company spe- 
cifically for experimental purposes for retarded adults 
attending a sheltered workshop in California. At the 
time of this trial run, our research population num- 
bered 15 retardates. Each was asked his reactions to 
this type of training upon completing the antennae as- 
sembly. The comments were unanimous in preferring 
the audio-visual method over conventional training 
approaches. The retarded remarked on the clarity and 
consistency of instructions and the excellent oppor- 
tunity it gave them to see exactly what was required in 
each job step. Many pointed out that different super- 
visors and foremen usually gave conflicting, inconsistent, 
and unclear instructions, while this was not so with the 
audio-visual presentation. In general, they reacted most 
favorably to the programmed audio-visual training 
approach, and were especially attracted by its mean- 
ingful presenntion. 

Developing an actual audio-visual program for one 
of the important skills required in Abilities Inc. was 
now our next step. It was decided to program the skill 
of plug soldering (or, more specifically, the skill of 
soldering wires to a cannon type connector). The 
soldering skill was chosen since it is one of the most 
important and eagerly sought after skills in an elec- 
tronics assembly plant. 

A small consultant staff of programmers and tech- 
nical assistants experienced in developing audio-visual 
aids and programs for industry met with the project’s 
staff to plan the plug soldering program for retarded 
workers. It was extremely important that these con- 
sultants be made aware of the retarded workers’ level of 



intellectual understanding. Moreover, our staff’s inti- 
mate knowledge of the skill to be programmed and the 
retardates’ functioning in an industrial setting was in- 
valuable knowledge in contributing to a successful pro- 
gram. Time and time again, the technical consultants 
required our professional staff’s recommendations be- 
fore an individual slide was photographed for final 
inclusion in the program. Similarly, in the writing of 
the audio script, the professional staff trained in mental 
retardation, had to edit and constantly revise the narra- 
tion so that it was meaningful to the retarded employee. 

Some of the principles that guided us in developing 
the audio-visual instructional program in plug soldering 
for the retarded worker included the following. 

First, the skill to be programmed had to be broken 
down into as many independent job steps as possible. 
A very thorough process of job simplification was car- 
ried out. This served as the foundation from which to 
build the program. 

Secondly, repetition of job steps was most important. 
For example, instructing the worker as to the proper 
manner in which to hold the soldering iron was repeated 
in the program every time a new pincup was to be 
soldered. Although a specific job step was presumably 
learned and being performed efficiently by the retardate, 
its instruction was nevertheless repeated in subsequent 
job steps whenever appropriate. It has been our experi- 
ence in teaching industrial skills to the retarded that 
such repetition combats forgetting or improper execu- 
tion of the skill. The practical application of this prin- 
ciple directly influences the length of the program. The 
total number of slides comprising this plug soldering 
program is 216 which is divided into 6 separate sub- 
programs or sections of 36 slides each. Each of these 
sections or sub-programs is devoted to one major aspect 
of the plug soldering job operation. 

Thirdly, verbal instructions need to be brief and 
easily understood. A knowledge of the retardate’s aca- 
demic level of achievement and his language usage was 
necessary when writing the script. Here, too, repetition 
was used. The same type of words and phrases were 
used over and over again. Sentences were as short as 
feasible. Technical words that are part of the operation 
programmed were never changed in any way. They 
were repeated and emphasized accordingly; in this pro- 
gram such words were inhibisol, flux, and solder. It 
should be noted that the narration was done by the 
training supervisor, the voice of a person with whom 
the retardates were already familiar. 

Lastly, the staff struggled with the problem of rein- 
forcement or feedback. There was no practical or 
economically feasible manner in which the equipment 
could inform the retarded worker whether or not his 
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response was a correct one. The feedback set-up would 
have had to be extremely sensitive and complex to 
accomplish this purpose. Moreover, the job skill of 
soldering does not require absolute type of responses, 
but allows a fine range of response variations that are 
still “correct” responses. This is the case for many 
industrial skills. The problem of feedback was resolved 
by having a training supervisor present while the re- 
tardates were following the audio-visual program. Since 
there will be no more than four retardates learning this 
skill at one time by the audio-visual program, a super- 
visor checking the work seemed the most feasible way 
to work out the question of reinforcement. When one 
considers that it takes approximately six months of con- 
ventional individual training and supervision to train 
a normal worker to become an efficient solderer, this 
use of one supervisor with the audio-visual program 
becomes acceptable from an economical and practical 
viewpoint. 

At this writing, the audio-visual program outlined 
has just been completed. Only preliminary trials have 
been run with some of the retarded workers. The 
project has not collected any substantial or definitive 
data regarding the efficacy of audio-visual programs for 
the retardate. This paper is in the way of a preliminary 
report. However, there are impressions and observa- 
tions that the author wishes to make at tMs time. 

One of the important values that this training tech- 
nique holds for the retarded lies in its ability to capture 
and hold the attention of the retardate. The problem of 
distractibility is evident in many mentally retarded. 
When those retarded workers who displayed difficulties 
in attention were exposed to the audio-visual program, 
this difficulty was greatly alleviated. They were more 



attentive and less inclined to look around and become ^ 

distracted when responding to the audio-visual training | 

teclmique. ^ I 

Another interesting finding has been that in pro- | 

gramming a skill for the retarded, the staff learned 3 

much about tlieir own training techniques and the jobs j 
being taught. This intensive involvement of analyzing j 
and evaluating one’s own personal resources while | 

progr ammin g forces one to leant and, in some cases, | 

modify professional approaches and concepts. Pro- | 
gramming a skiU certainly helps the teacher learn from I 
his techniques. 1 

It is felt that progranuned audio-visual training may | 
have significance as aids in orientation and production, | 
in addition to its training function. Once this project’s | 
training data has been evaluated, the author plans to | 
develop audio-visual programs relating to teaching the | 
retarded basic knowledge about tools, equipment and | 
typical work or factory procedures and regulations. Its | 
value and potential as a production pacemaker will also | 

be explored. Once a worker has learned a job skill and | 

is on the production line, the feasibility of pacing his | 
production by audio-visual methods is an interesting ^ 
possibility. I 

It should again be stressed that this paper is merely | 
a preliminary report on the thinking and developments, j 
to date, of a pioneer project that is using programmed | 
audio-visual methods of training the mentally retarded I 
industrial job skills. The staff of the project does not | 
envision audio-visual training techniques as a replace- | 
ment of the individual supervisor, but as a training aid | 
that has a tremendous potential for the retarded. That J 
this potential can be more clearly defined is one of the | 
project’s major goals. | 



LINGUISTICS, LANGUAGE, AND DYSLEXIA* 

Rudolph C. Troike 

After a long and systematic discussion of the nature of language writing, language 
development, and reading and reading disability (dyslexia), Rudolph Troike points 
out that programmed instructional techniques may yield results through syntactic struc- 
tural drill in the treatment of reading deficiencies that have remained uncorrected by 
other remedial techniques. Programmed instruction seems ideally suited to the care- 
fully controlled, but motivating presentations and repetitions of materials that are 
needed to condition the child to the phonemic-graphemic correspondences of English. 
The reader can draw his own conclusions for its usefulness in other languages. 



It seems as appropriate as it is unusual to have a 
linquist participate in a symposium on the subject of 
reading disability (dyslexia). Man is the symbol-using 
animal par excellence and language is the symbolic 
system which forms the basis for much of human cul- 
ture and social interaction. As the science which con- 
cerns itself most directly with the nature of this 
symbolic system, linguistics has a unique and vital con- 
tribution to make toward a better understanding of 
language disorder and symbolic dysfunction in general. 

Linguists, as linguistic scientists are known, have 
concentrated their work on the analysis of language 
structure, in an effort to discover the formal mecha- 
nisms used by speakers of a language for com- 
munication. Questions of speech development and 
language dysfunction have thus far been of only periph- 
eral concern to linguists. But linguistics holds the 
promise of a major contribution to the elucidation of 
these matters because of its emphasis on the funda- 
mental structure and operation of language. An un- 
derstanding of the nature of linguistic phenomena is of 
basic importance to the dvelopment and implementa- 
tion of corrective treatment for language disability. 

We may begin by considering some of the terms in 
a general definition of language. In the broadest 
sense, language is any system of arbitrary, conven- 
tional symbols used for communication. The demands 
of communication require that the use of symbols be 
organized into a patterned system; their occurrence 
cannot be entirely random and unpredictable. Thus, a 
sequence of dance gestures, or the dance of the honey- 
bee, or the utterance “The dog bit the man,” each 
manifest an underlying system of recurrent patterns 
of relationship among their constituent elements. The 
basic units of language are arbitrary, conventional sym- 
bols: symbols because the units “stand for” their 
referents; arbitrary because there is no inherent con- 
nection between the symbol and its referent- and con- 
ventional because it is essential that the referent be 



* Editor’s note: Even though this paper has little direct con- 
cern with programmed instruction it was felt by the editors 
that the material presented had strong implications for the 
instructional technology. 



known and agreed upon by those who use the symbol. 
Thus, the word horse meets the conditions of this defi- 
nition. The sounds used in this word bear no real con- 
nection to the animal, Equus Caballus; but as used by 
English speakers, there is common agreement as to 
what they refer to. To speakers of other languages, 
this sequence of sounds carry no such information. 
Similarly, for a speaker of En^ish to use horse to refer 
to a dog (Canis familiaris) would be to invite confu- 
sion; consider, for example, the reaction to such a 
statement as “I keep my horse in the house.” In order 
for the meaning of a message to be successfully trans- 
mitted, therefore, both the sender and receiver must 
be parties to the same set of agreements as to the value 
of the symbols and the system in which they are used. 

The acoustic signals used in speech are organized 
into a system of contrastive sound-units for which the 
term phoneme is used. We know that the speakers of a 
language do not react to “pure sound” as such, but are 
conscious only of those distinctions in sound which 
function to differentiate words in the language. A native 
speaker has trouble hearing or recognizing sub-phone- 
mic (non-contrastive) sound variations such as that 
found in the two “p-sounds” (bilabial stops) in pin 
and spin. This correlates with the observation that the 
child’s acquisition of the neuro-muscular control re- 
quired to produce this sub-phonemic difference in 
sound seems to be an unconscious development. 

The concept of the phoneme enables one to predict 
learning problems from the speakers of a given lan- 
guage who undertake to learn any specific second lan- 
guage. For example, a person with a Spanish-speak- 
ing background who fails to make or hear a distinction 
between English ship/sheep, etc., does not have a 
speech or hearing defect, but is simply reacting in 
terms of his native Spanish sound-system, which has 
only a single phoneme /i/ where English has two, 
/I/ and /i/. As will be seen in a later section, an 
understanding of the phoneme also serves to clarify 
certain prevailing misconceptions about language de- 
velopment in children. 

The grammatical system specifies the relationships 
obtaining among the vocaubulary elements. Part of the 
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“meaning” of any utterance is given by its grammat- 
ical structure. ITiis may be shown by several exam- 
ples: 

( 1 ) John hit Bill vs. Bill hit John. 

Despite the fact that the same words are used in both 
sentences, they do not have the same total meaning. 
The difference in meaning is due to the underlying 
sentence pattern actor-action-object which determines 
the significant relation of the words to one another. In 
English, the syntactic device of relative word order is 
important, because it signals vital information of the 
type just illustrated, as well as other types; e.g.: 

(2) This is a house guest vj. This is a guest house. 

(3) He is still standing vs. He is standing still. 

Thus we see that for English at least, we cannot simply 
add up the separate meanings of the individual words, 
as if they were numbers, and expect to arrive at the 
total meaning of the sentence. An important part of 
the meaning is contributed by the implicit structure 
of the sentence pattern. 

(4) The ^-s ^-ed ^-ly 

(Adj.) (Noun) (Verb) (Adv.) 

in the rish 

(Adj.) (Noun) 

In addition to word order, structural relations may 
also be signaled by junction words (such as the, a, 
in, to, of, etc.) or word affixes (inflectional suflBxes: 
-s, -ed; derivational suffixes: 4y, -ish). Each of these 
three grammatical devices may be more or less im- 
portant, or even absent, in other languages. In the ex- 
ample above, only the sentence skeleton is shown. The 
blank spaces may be filled in with various words to 
produce many different sentences, but only a certain 
class of words (shown by the part-of-speech identifica- 
tions) may be used in each particular blank. The infor- 
mation signaled purely by the grammatical framework 
of the sentence may be teimed “structural meaning.” 

The types of words which would fit into these blanks 
to form meaningful sentences may be called content 
words to distinguish them from function words. Con- 
tent words make up the vocabulary and provide the 
principal content, or “lexical meaning,” of utterances. 
As the vocabulary of a language reflects the content of 
the culture, it is an important element in the encultur- 
ation of the child. Nevertheless, it is too easy to over- 
rate the importance of vocabulary and ignore the much 
greater significance of grammar. The grammatical 
system is the real heart of the language, and its mas- 
tery is a prerequisite to successful cultural learning 
and social training. 

It is important to realize that language is a social 
phenomenon, part of man’s cultural behavior. The 
langauge which any individual uses is the result of his 
conditioning to the system used by others around him. 



Like other aspects of culture, language changes con- 
stantly with respect to time and space. A group of 
people who use a single undifferentiated system may 
be termed a speech co mmuni ty; two speech commun- 
ities exhibiting obseiwable linguistic differences in vo- 
cabulary, grammer, or pronunciation, but whose 
speech is mutually intelligible, may be said to have dif- 
ferent dialects. 

Dialects arise as a natural, product of frequency of 
communication within and between geographical areas 
and social strata. In any language used by more than 
a sin^e in-group, there will be dialects. In the United 
States at the present time there are several regional 
standard dialects which are the product of our history 
of settlement and westward expansion. While recogni- 
tion of these facts forces abandonment of the myth 
of “‘pure” or “correct” English (or Spanish or Ger- 
man, etc.), an awareness of the sociological status of 
different dialects and styles provides an objective basis 
for determining the goals of language instruction. The 
regional or local prestige dialect, rather than some ab- 
stract set of absolute rules, should form the model to- 
ward which a teacher should intelligently guide her 
pupils’ speech-ways. 

Widespread misunderstanding of the relation or writ- 
ing to speech and language has proven a severe 
obstacle to the development of adequate reading ma- 
terials and programs in American schools. This mis- 
conception arises from the naive and false assumptions 
that writing, because of its permanence, is more basic 
than speech, and that written English, is the language, 
from which the imperfect usages of speech derive. 
This fallacy often finds its expression in such phrases 
as “. . . the sound of the letter h.” 

It must be stressed that language is basically spoken 
and not written. Writing is a secondary or derived sys- 
tem of symbolization which attempts to represent the 
phonemes of a language in a visual, preservable form. 
Despite the enormous importance of writing in our civ- 
ilization, the course of language development would 
scarcely be affected if all writing were to disappear 
from the earth. 

A realization of the true nature of writing is essential 
to a proper understanding of the reading process and 
the preparation of special reading materials. The child, 
who is already equipped with a speaki.ng knowledge of 
the language, is taught to associate his speaking habits 
witli a set of visual symbols. Under ordinary circmn- 
stances, he learns to decode the visual stimulus into 
its spoken referent (language), then decode the lan- 
guage stimulus for its information content. Reading is 
thus a two-step process, and reading skill is, to a large 
extent, dependent upon the mechanical facility and 
speed with which the child is able to perform it. 
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The notions that the printed symbols literally contain 
sounds (which the child “gets out” of them), or that 
the reader goes directly from the visual stimulus to the 
idea which it represents, both represent misleading fal- 
lacies which continue to impede the adequate teaching 
of reading. For children with any degree of involve- 
ment with dyslexia, these fallacies and their attend- 
ant teaching philosophies may impose additional or 
insurmountable handicaps to the mastery of reading. 

Studies carried out by Irwin and others at the State 
University of Iowa have established two periods of 
prelinguistic behavior in children. The first, which ex- 
tends from birth to about the third month, is marked 
by the reflexive sounds associated with physiological 
states. Considerable attention has been given to the 
types of sounds produced by the infant during this time. 

From the third month through the eighth month, 
apparently as a result of maturational development, 
the child goes through a period of babbling, in which 
the sounds are produced primarily as a result of auto- 
stimulation. A profusion of sound-types are produced, 
including many which will not be needed for the later 
language. 

Toward the end of the first year the child begins a 
transition from an egocentric to a social orientation. 
This period, which may be termed the proto-linguistic 
stage, finds the child beginnig to react to words and 
sounds produced by those around him, by imitating 
them. From his first birthday to about eighteen months, 
he learns individual words as true symbols, and builds 
up an active vocabulary of about twenty items. He 
also develops a new type of vocal behavior termed 
variously “jargon” or “jabbering.” 

Following the eighteenth month, and concomitantly 
with maturation of Broca’s area in the cerebral cortex, 
the child enters upon the formative phase of the true 
linguistic stage in learning. Some recent research on 
this phase (Braine 1963, Glanzer 1962) has shown how 
the child develops a rudemintary grammar involving 
two-word sequences, and gradually expands and dif- 
ferentiates this until his utterances develop a gram- 
matical structure closely matching that of speakers 
around him. Seemingly the child possesses an innate 
capacity for forming inductive theories about gram- 
matical structure (Teeter 1963, Chomsky 1962), and 
the capacity perseveres until about the onset of puber- 
ty. Up until this tin. , the learning of a second lan- 
guage can be accomplished with much greater ease 
and rapidity than at any time thereafter. 

From the time a minimal grammatical structure 
is achieved, through the sixth or seventh year, the 
child passes through what might be called the develop- 
mental phase of the linguistic period. This is a time of 
continual practice on the basic grammatical pat- 
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terns and the acquisition of additional vocabulary. 
Smith (1935) estimates that children of 3 to 4 years in 
age speak over five million words a year. With the 
achievement of an adequate degree of linguistic com- 
petence and sensory-motor maturation, tihe child is 
prepared to begin formal schooling, wherein he will be 
instructed through the medium of language and will 
learn the difficult and complex skill of reading. 

Although the question of “reading readiness” is still 
insufficiently explored and fraught with folklore, lin- 
guistics can provide some important insights into the 
matter. Beginning with the proto-linguistic stage of 
language development and continuing tlirough the de- 
velopmental phase of the linguistic stage, the child 
gradually learns to recognize and produce the phone- 
mes of his native language. This is a gradual proc- 
ess, and may extend into the sixth or seventh year. 
However, children with severe hearing loss, aphasia, 
or mental retardation may never learn all the phonem- 
ic distinctions of their parental language. 

In general, it may be said that a child has not mas- 
tered the phonemes of his language until he is able 
to consistently control their use in the pronunciation of 
otherwise identical words (e.g., r^in/5in), and react 
to them appropriately. The random production of a 
given sound during the earlier stage of babbling has no 
relation to its use by the child as a phoneme in the later 
language. 

Since reading involves the association of visual stim- 
uli with the phonemes and phoneme-sequences of the 
native language, reading-readiness involves as a pre- 
requisite the child’s mastery of the phonemic system of 
his language. If the child has not learned to differen- 
tiate the pronunciation of /w/ and /r/, for exam- 
ple, he will not be able to determine which symbol 
represents which sound. Since English orthography is 
largely phonemic (i.e., there are regular correspon- 
dences between the printed symbols — ^graphemes — and 
the phonemes) , the failure of the child to hear certain 
sound distinctions will increase the burden of learning 
to read, and will inevitably contribute to his confusion 
and frustration. 

Dialect differences sometimes cause problems and 
lead to misunderstanding, especially if the teacher and 
pupil are from different dialect areas. The failure of 
speakers in large areas of the United States to differ- 
entiate the vowels in pin/pen, tot/taught, and hoarse/ 
horse, or the initial consonants in which/witch, should 
be recognized, and children should not be criticized or 
penalized for this by forcing them to produce distinc- 
tions which are foreign to their native dialect. 

The large residue of English spelling which is not 
phonemic has to be approached by different teaching 
techniques. There are very few words whose spelling 
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bears no relation to their pronunciation, but there are 
many which bear only partial relation. In addition, 
there are problems of multiple representation of the 
same sound, as well as identical representation of dif- 
ferent sounds. When the phoneme '.letter correspond- 
ence does not hold, the child must be taught to memo- 
rize individual wordiletter-sequence correspondences. 
The amount of such information which an individual is 
capable of storing seems to vary widely, but well-or- 
ganized materials and teaching techniques should serve 
to bring the child closer to his maximum capacity. 

Reading disability (dyslexia) represents a specific 
manifestation of a more general disorder, that of sym- 
bolic dysfunction. Dyslexia refers to the inability to 
associate visual stimuli and linguistic responses. This 
inability may be inherited or acquired; it may be due 
to physiological, functional, or psychological causes; 
and it may occur independently (primary dyslexia) or 
in conjunction with other types of symbolic dysfunction, 
such as aphasia (secondary dyslexia). It is also im- 
portant to note that individual involvement may range 
from marginal to complete. 

The tremendous premium placed upon literacy in our 
modem society emphasizes the significance of read- 
ing ability as a culturally important skill. Children 
with primary dyslexia may be helped by a linquistic- 
ally sophisticated teaching program and appropriate 
teaching materials. In severe cases, where no compli- 
cations are involved, the child may be helped by sup- 
plementary oral instruction and drill in much the same 
way that blind students are enabled to go through 
school and college programs with the tid of “reading as- 
sistant.".” Emphasis on the development of oral recall 
ability would perhaps enable such individuals to par- 
tially overcome their handicap. Certainly, many teach- 
ing procedures associated with both the “phom'cs” and 
the “look-say” approaches to reading instruction are 
deleterious to children with even minimal dyslexic in- 
volvement. 

One of the difficulties in the treatment of dyslexia 
is the failure of teachers to diagnose the problem soon 
enough. Where dyslexia occurs together with or as a 
reflex of other language-development problems, the 
child may be designated for special treatment from 
an early age. When it occurs alone, however, it will 
not become apparent until reading instruction begins, 
and may be difficult to differentiate from cases of 
slow learning. Delay in diagnosis prolongs the frus- 
tration and embarrassment of the child in his reading 
experiences, and renders his problem increasingly 
intractible of treatment. 

Although the general outlines of language ontogeny 
in the child are known, detailed descriptions which 



would permit the early diagnosis of aberrant develop- 
ment are not yet available. Research currently in 
progress, based on the linguistic model for the descrip- 
tion of syntactic structure d ,^eloped by Noam Chom- 
sky (1957), promises to yield important data in the 
near future. Previous investigations in this area have 
not had enough linguistic sophistication to provide pre- 
cise, meaningful results. 

While indices of gross vocabulary size provide some 
indication of the language development of the child, 
more significant indices based on the rate of syntactic 
development can be devised when these studies have 
been completed. The child’s mastery of the structural 
devices of his language should provide the most reliable 
indicator of his linguistic development. 

As mentioned previously, most children appear to 
have a native capacity for inductii'e theory formation 
with respect to language. Where this faculty fails to 
function normally, it may be that carefully designed 
structural drills — such as those used for teaching 
English as a second language — ^may be employed for 
remedial instruction. These could be started fairly 
early, and their use might help to compensate for the 
child’s deficiency by forcing pre-analyzed patterns on 
his attention. This is clearly an area in which program- 
med instructional techniques, coupled with linguistic- 
ally adequate materials, could be usefully applied. 

In cases of simple primary dyslexia, programmed 
instruction seems ideally suited to the carefully con- 
trolled, but motivating, presentation and repetition of 
materials that is needed to condition the child to the 
phonemic: graphemic correspondences of English. Or- 
dinary classroom teaching does not provide the in- 
tensive conditioning and linguistically-devised materials 
which may be necessary to overcome the handicap of 
dyslexia. 

The problems of language disability have been ex- 
tensively studied by neurologists, psychologists, and 
speech pathologists, and from their work an important 
body of information relating to the subject has been 
amassed. Much previous work, however, has been 
weakened, or even vitiated, by insufficient knowledge 
on the part of the investigator regarding the structure 
of English, the nature of language, or the relation of 
writing to speech. Specialists in the study and treat- 
ment of language disorders have pointed out the need 
for closer collaboration with linguists to provide a bet- 
ter understanding of the substantive features of lin- 
uistic systems. It may be hoped that the awareness of 
this need will become mutual, and that linguists will 
contribute their knowledge and experience to help 
provide a truly multilateral attack on this grave and 
difficult social problem. 



Vi industrial Applications 

AEROSPACE INDUSTRIAL PROGRAMMING (PAST, PRESENT AND FUTURE) 

Jac D. Meacham 

Jac Meacham briefly reviews a few of the aerospace training situations in which 
programmed instruction has been and is currently being used. The use of programming 
will undoubtedly continue in the futuic. Jac Meacham feels that the automated instruc- 
tion will also prove useful in the area of aerospace system analysis and design. 



Aerospace pertains to the earth’s envelope of atmos- 
phere and the space above it, two separate entities 
considered as a single realm for activity in launching, 
guidance, and control of vehicles which will travel in 
both realms. Aerospace industries are those companies 
that are actively engaged in furnishing products and 
services for aerospace forces. Witli the increased com- 
plexity of aerospace equipment, the problem of com- 
munication and training becomes extremely important 
and a constant vigil must be maintained to provide the 
using personnel with adequate and up-to-date informa- 
tion. The technical information required for aero- 
space products and services is no small item and it far 
exceeds that of all the combined publishers of com- 
mercial material in the country, the technical infor- 
mation required by the users of aerospace equipment 
must be: Understandable, accurate, complete, and 
usable under the most difficult situations. 

Decentralization of the producer of aerospace data 
and the user has compounded the situation to the point 
where something nearly revolutionary must take 
place in order to reduce the danger of our becoming 
technologically inadequate and insufficiently qualified 
to perform the demanding tasks of space-age existence. 
What does all this have to do with programmed instruc- 
tion? 

I feel that a solution or partial solution to the vast 
problems of aerospace communication and training lies 
in programmed instructional materials. The need is 
great for uniform standardized aerospace technical 
data and control of the material presented. This can, 
and is, being accomplished via teaching machines and 
programmed instruction concepts. Aerospace indus- 
tries are, or should be, faced with the task of prepar- 
ing this material and they must be prepared to increase 
their ability to produce good quality programs. The 
ultimate objective as I see it, is to produce program- 
med instruction that would result in uniform interpre- 
tation and understanding, and maximum utilization for 
the Shorty Powers’ and John Glenn’s of tomorrow . . . 

Both the aerospace industry and programmed in- 
struction, despite their revolutionary characteristics, 
are additions to oui recent past. The aerospace indus- 
try’s past is well known to most, but the programming 



aspects of this dynamic field may not be. The begin- 
ning of programmed instruction as a part of the aero- 
space training curriculum actually began to take shape 
in an underwater project, the atomic submarine. 

A handful of men went to work on a unit called a 
“teaching machine.” The ultimate device actually 
turned out to be more of a “guidance machine” in that 
most presentations guided the worker in accomplish- 
ing a procedure, rather than teaching him the proced- 
ural steps. The elforts of others began to show up in 
the research and development phases. These pioneers 
showed early success in the fields of shipboard missiles, 
supersonic airplanes, and other military training areas. 

Not loo long ago, the aerospace industry was faced 
with a requirement which assured a schedule and plan 
of completion for events and activities to be used on 
aerospace projects. The result was a concept called 
PERT (program evaluation review technique). Thou- 
sands of people needed to use the system, many fur- 
nished input to the system, and still others were re- 
quired to coordinate and develop the web of informa- 
tion. A training problem had to be solved before PERT 
could be implemented. Several organizations started to 
program the material and at least two successful pro- 
grams are now available for aerospace contractors. 
Today, across the nation, widespread and comprehen- 
sive PERT training can take place in less than an 8- 
hour work day. 

Another page out of the past demonstrates the use 
of programmed instruction in the operational aspects 
of aerospace products. KIFIS (Kollsman Integrated 
Fliglit Instrument System) corrects, combines, modi- 
fies, and refines information from aerospace instru- 
ments and provides the pilot with additional informa- 
tion to make his job easier. Extensive experience in a 
classroom situation indicated that the subject was 
difficult to teach due to variable understanding among 
the students. It was decided that programmed instruc- 
tion might take care of some problems of boredom, ab- 
sorption, and pacing that existed. The results were 
not astoUiShing but did show that this type of training 
was accomplished faster and with greater student ac- 
ceptance and understanding. 
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Other efforts in the past, that have produced some of 
the well known benefits of programmed instruction, in- 
clude: 

1. Safety Precautions for Nuclear Weapons — a pro- 
gram designed for crew members v/ho are responsi- 
ble for loading weapons on the supersonic high 
performance aircraft that protect our nation. This pro- 
gram teaches the aspects of personnel safety as per- 
tains to the weapon and the weapon system. 

2. The Rocket Package Test Adapter — a program 
designed to meet the needs of loading crew members, 
and contains information on how to operate and main- 
tain the Test Adapter. 

3. Flying Safety Views — a program that discusses 
the controversial problem of ejection vs. forced land- 
ing and provides the pilot with the knowledge that will 
enable him to make a decision based on fact. In passing, 
we also note The Arrest Hook System for the F-102 
aircraft, the Value Control program, the Ejection Seat, 
the F-106 Hydraulic System, soldering, test and launch 
stand operation and maintenance, task analysis and 
many many other efforts that are part of the past aero- 
space programming work. The success of these early 
programs have demonstrated the desirability and 
effectiveness of programmed instruction as a potential 
solution to the vast training and instructional problems 
of aerospace technology and related fields. 

- The armed forces are showing much interest in the 
field of teaching machines and programmed instruc- 
tion. The Air Force, with its two-phase attack, is cur- 
rently going from development of programmers and 
programs into direct application. Sage, Electronics, 
computers, and other military oriented programs are 
being utilized daily by Air Force personnel at home 
and abroad. The Navy is utilizing programmed in- 
struction for seamen in the maintenance and opera- 
tion of shipboard missiles. Great Lakes Naval Train- 
ing Center and other training and testing sites are 
installing programmed materials for their students. 

' Several large aerospace contractors have actively 
engaged in the use and development of programmed 
materials. These aerospace oriented fiurms are pro- 
gramming for in-house use and for military users. 
.Weapons and weapon systems are being studied. 



These include among many others missile compo- 
nents, operation, guidance, launching, recovery, safety, 
overhaul, storage, and retrieval problems. The com- 
mercially available, academic, oriented programs are 
also being utilized by aerospace industrial and military 
persoimel to provide a foundation or springboard for 
more complex systems and procedural programs. 
Off-the-shelf programs for aerospace users are very 
limited and many courses are needed. 

Present use of programmed materials in the aero- 
space industries show that programmed instruction 
can: 

Accelerate training. 

Permit training and retraining on a non-scheduled 
or remote-site basis. 

Release instructors for advanced and supplemental 
duties. 

Relieve the critical shortage of qualified instructors. 

Standardize complex technical instructional material. 

Future use of programmed instruction within aero- 
space industries is being assured through research and 
development of specifications for programmed instruc- 
tion in FASTI, Astro-Aid, CLASS, and many more 
subjects. Programmed instruction for aerospace in- 
dustrial use will continue to grow with the complexity 
of the equipments and lack of qualified and weU trained 
personnel necessary to operate and maintain them. 
The procedures for operating and maintaining equip- 
ment MUST be eflScient, reliable and safe. Scientific 
knowledge concerning human performance will be 
applied for the purposes of: 

a. Achieving system performance by appropriate use 
of the man as a system component. 

b. Developing, selecting and designing equipment, 
procedures, and facilities to insure efficient, reliable 
and safe human performance within the specified tol- 
erances. 

c. Reducing demands on manpower resources, 
skills, training, and procedural data within the estab- 
lished parameters. 

TTiese and other factors directly offer the program- 
med instruction field as a solution to the problems that 
are facing aerospace industries today for tomorrow’s 
space flights. 



I 



INDUSTRIAL PROGRAMMED INSTRUCTION IN VALUE CONTROL TRAINING 

Charles M. Keys 

Charles M. Keys reports on a programmed instruction package which was developed 
for use in Value Control or Value Analysis training of industrial employees. This pro- 
gram effected a significant reduction in the training time over the conventional course 
without a loss in training quality. In addition to a savir.gs of nearly forty percent in 
training time, value control could be applied to a Value Analysis Course. 



Industry today is faced with intensive competition 
both from within the United States and from other 
countries which are challenging the superiority of the 
American industrial genius. Competition is keen from 
within our country because of the tremendous technol 
ogical advances in recent years and the cold, hard fact 
is that West Germany, Japan, France, Italy, and Swe- 
den, to name a few, are succeeding in competing with 
us for the world trade dollar. 

In an effort to meet this challenge, industry is pressed 
with the need to make all employees more cost con- 
scious than ever before. This applies especially to 
employees who are placed daily in positions which re- 
quire them to make cost decisions. One management 
tool which has been developed in recent years to give 
industrial employees a specification for making cost 
decisions is “Value Control”. 

Value Control is also known within industry as 
“value engineering” and “value analysis.” Its design is 
to shape the performance of industrial employees in 
such a marmer that they will make cost decisions which 
will result in the best value for the dollar. Very briefly. 
Value Control is the systematic application of certain 
techniques which: 

1. Identify the function of a product of service 

2. Establish a value for that function. 

3. Endeavor to provide that function at the lowest 
total cost without degradation. 

Unnecessary costs are eliminated and suitable alter- 
natives are developed. While the phases of the Value 
Control process are neither new nor revolutionary, they 
are tied together in a neat package and presented in 
a seminar/workshop training program to create a 
cost-conscious atmosphere. 

The importance of a value control training program 
is pointed out by the Federal Government’s attitude to- 
ward the subject. In recent years, value engineering 
“incentive clauses” have appeared in many Govern- 
ment contracts. Recently, value control has become a 
mandatory requirement in some types of Government 
contracts and is provided for in the Armed Services 
Procurement Regulation. 

The decision to convert the Value Control Training 
program to a programmed instruction course was a 
simple matter of applying value control to Value Con- 
trol. The training course was studied to determine how 



it could be streamlined to reduce costs and increase 
effectiveness. Programmed instruction was the an- 
swer. It was apparent that an internal training situa- 
tion could be improved simultaneously with the devel- 
opment of a marketable product. Hence, under “con- 
tract” from the Training Department and with the full 
support of the Manager of Value Control, the Convair 
programmed instruction group attacked the project. 

The subject of Value Control lends itself to division 
into five fairly distinct parts. This division conformed 
very well to the requirement for conducting a work- 
shop concurrently with the presentation of the subject 
matter in the actual training situation. The five seg- 
ments which were programmed were: 

Orientation (an overview of the subject) 

Identification (of the function) and Information 
(gathering) 

Speculation (developing alternatives) 

Analysis and Selection (of the best alternative) 
Presentation (selling the alternative) 

Since the “customer” was serving as subject matter 
expert, the development of content outlines and course 
objectives was somewhat simplified. The guidance 
provided by our Value Control Coordinators was rea- 
sonably specific and the limits of coverage were well 
defined. 

The programming task, however, developed into 
much more than a simple conversion of an existing 
course to programmed format. The lecture course, as 
it had been taught, had no content examination and 
little printed study material. There was no text. The 
lectures were given by qualified personnel from many 
different areas of the company. While this method 
was effective, it was also somewhat cumbersome and 
required extensive coordination with many people 
prior to and during each seminar. This same extensive 
coordination was required by the programmers in 
accumulating subject matter and source data. 

As previously stated, the curriculum content was di- 
vided into five parts. Accordingly, a five-man team 
was used, with each programmer assigned the re- 
sponsibility for development of one of the parts. When 
a team is used, it necessitates close coordination be- 
tween the programmers, particularly during the initial 
development of content outlines and specification of 
objectives. Careful and frequent editing of programmed 
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material is required to prevent duplication and over- 
lapping coverage. 

A team project also has some advantages. Primary 
among these is the fact that all members gain ex- 
tensive knowledge of the entire subject matter. Each 
programmer can then edit the material of the other 
team members and provide suggestions for improve- 
ment of subject matter content as well as program- 
ming methodology and technique. This is quite fruit- 
ful, especially when the group is compatible and the 
project is a bonafide team effort. 

Because of the nature of the subject matter, a branch- 
ing technique was used. Some “linear review” of key 
material was used, but, for the most part, the require- 
ment was that the student discern a correct response 
from a group of two, three or four possible alternatives. 
In addition to the extensive editing by the team mem- 
bers and project leaders, first and second drafts were 
tested on approximately twenty typical subjects to 
determine deficient and weak areas. 

The next phase of development consisted of testing 
the drafts in a formal seminar consisting of twenty 
students. The texts were used in conjunction with a 
workshop project with some experimentation to deter- 
mine how best to fit them into the workshop phases 
of the actual training situation. 

After a careful review of the students’ accomplish- 
ments and with additional feedback from the Value 
Control coordinators, minor revisions were made to 
40% of the 350 frames and the first edition was pub- 
lished. This edition was used in formal seminars on 
forty students and minor revisions were made prior 
to publication of a revised edition which is currently on 
the market. 

The results of the final testing of the programmed 
texts were most gratifying. The total course cost and 
classroom time were reduced by more than twenty- 
five per cent. A valuable asset was gained in the 
improvement of the flexibility of class organization 
and presentation of material. 

As previously discussed, the lecture course had used 
guest speakers who held key positions througout the 
plant. There was approximately twenty-four such lec- 
tures which not only required the speakers’ time but 
necessitated large classes (thirty to forty students) in 
order to justify the time spent by the speakers. 

With the development of the programmed texts, the 
classes may be much smaller (fifteen or twenty stu- 
dents) offering a better opportunity to perform on 
workc^op projects. These projects require that the 
classes -vontact many departments in the plant and 
with the smaller classes- the persons contacted can 
give each team of students more time and consequently 
better information. 
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The following table shows a comparison of the con- 
ventional training methods versus the programmed 
texts. 



Conventional Training Programmed 
Methods (Lectures) Instruction 
(N = 50) (N = 50) 


LECTURE TIME 


10 Hours 


30 Minutes 


READING TIME 


30 Minutes 


6 Hours 


TOTAL CLASS- 
ROOM TIME 


10.5 Hours 


6.5 Hours 


EXAMINATION 

RESULTS 


No formal 
exam given 


90% Correct 


WORKSHOP 

RESULTS 


Students in both workshops de- 
monstrated equal ability in ac- 
complishing the phases of value 
control 



In addition to full-scale use in the Convair Value 
Control training program, the programmed texts are 
being used by other General Dynamics divisions. They 
are also being used by San Diego City administrators 
and by the U.S. Navy in San Diego. There have been 
numerous inquiries from other potential customers. 

When a project is completed and the “gains” and 
“losses” are tabulated, it is most gratifying to see in 
retrospect that more has been accomplished than was 
initially anticipated. Such was the case with Value 
Control. Many “side-effect” benefits may be realized 
by the conversion of an existing industrial training 
course to programmed instruction. Looking at the 
additional benefits at this time, it is easy to say that 
they should have been anticipated. The bonus received 
was in the form of the following; 

1. Conversion of an existing industrial training 
course to programmed format forces you to take a close 
look at your training material. It offers an excellent op- 
portunity to revise and update the subject matter and 
reorganize it around a tight outline to improve con- 
sistency of training. 

2. The conversion causes you to more clearly specify 
your tr ainin g objectives and determine if training time 
is really being used to the best advantage. When you 
start the change to programmed instruction you should 
ask, “Are the students in the existing course learning 
what they must know in order to perform the duties 
for which they are being trained?”. Do not assume that 
the existing course is doing this. 

3. Converting a course such as Value Control to pro- 
grammed format creates a renewed interest in the 
subject within an organization. In this case, the entire 
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Value Control program was given added importance 
because the training program was doing something new 
and was being widely discussed and publicized. The 
training effort was in the limelight because of the “ex- 
periment” and there was increased enthusiasm at the 
top management level and throughout the ranks. 

4. Although anticipated and not really a side-effect 
benefit, one additional asset gleaned from the conver- 
sion was the enhancement of the overall programmed 
instruction effort. The programming group gained 
valuable experience and flie concept of programmed 
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instruction was publicized and more widely accepted 
throughout the company. The boost to programmed 
instruction was so great that it accelerated our efforts 
into new areas of endeavor. We are currently pro- 
gramming quality assurance training courses for in- 
house use on such subjects as Material Review Board, 
Nondestructive Testing, Traceability, and other hard- 
ware inspection techniques. This new effort would 
probably not have been as readily accepted had it not 
been for the success of the initial endeavor. Value Con- 
trol. 



TECHNOLOGICAL CHANGE AND THE JOURNEYMAN ELECTRICIAN: 
AN EXPERIMENTAL STUDY IN CONTINUING EDUCATION* 

David S. Bushnell 

David S. Bushnell outlines the problems associated with adult education and de- 
scribes a research program which utilized programmed instruction and teaching 
machines in this field of teaching. The study, conducted by Stanford Research Institute, 
taught journeyman electricians the fundamentals of electricity using programs only, a 
combination of programs and teachers, and teachers only. The study showed that the 
use of teaching machines combined with teachers reduced training time, motivated 
the student, and allowed for individual differences through self-pacing. Mr. Bushnell 
concludes his report by proposing several areas for further study. 



Last year over 26 million adults, one out of every 
five in the United States, participated in some form of 
educational activity. Almost any organized group that 
you could name, industries, churches, professional or- 
ganizations, sponsors one or more programs of adult 
education. Most of this adult learning was of the non- 
credit type and was concentrated most heavily within 
the areas of vocational and recreational subject matter. 
According to a recent report by the National Opinion 
Research Center (Johnstone, 1963), one-third of all 
adult education studies are in the vocational sphere. 

With increasing amounts of leisure time and greater 
specialization of work skills, the emphasis on vocational 
education promises to increase within the next few 
years. Initial employment skills, once thought to be 
all that was needed to maintain a person throughout 
his entire career, must now be upgraded or revised 
several times to adapt employees to technological ad- 
vances. The retraining of workers whose skills are in 
danger of becoming obsolete has become an impor- 
tant part of the national effort to maintain the current 
level of employment and rate of technological growth. 

But training effectiveness, particularly in those in- 
dustries which rely upon voluntary participation, has 
been hampered by at least three factors: (1) the im- 
willingness of adults to enroll in and remain enrolled 
in voluntary training programs, (2) inadequate adult 
teaching techniques, and (3) a shortage of qualified 
instructors. For many skilled workers the prospect of 
retraining is impleasant because of past educational 
experiences; some are early high school dropouts; 
others have been out of school for 20 or more years; 
others because of their previous experience have found 
courses to be too difficult or not appropriate to their 
particular job needs. 

There is an urgent need for new ways to motivate 
adult workers to enroll in appropriate voluntary pro- 
grams and to continue once they are enrolled. Improv- 



* Study sponsored by The Bureau of Industrial Education, 
Calif. State Dept, of Education arid The Electrical Construc- 
tion Industry of San Mateo County. 



ing the appeal of adult vocational training may hinge 
in part upon the use of new teaching techniques that 
permit the student to proceed at his own rate of learn- 
ing without fear of failure; and in part upon recogni- 
tion of the fact that adults differ from high school or 
college students. 

Recent research has demonstrated that teaching 
machines are as effective as live instructors in impart- 
ing knowledge. The employment of auto-instructional 
techniques has reduced IN HALF the time required for 
many types of training programs. The real promise 
of teaching machines, however, may be their ability 
to overcome adult reluctance to expose themselves to 
the conventional classroom and to possible failure in the 
eyes of their associates. 

Some of us at Stanford Research Institute (SRI) 
have been experimenting with teaching machines, 
not in the area of adult vocational education, but in the 
field of postgraduate education of dentists and doctors. 
We found that self-instructional procedures, or auto- 
instruction, appealed to this caliber of student because 
it allowed him to move at his own rate and almost 
guaranteed successful learning. The programmed 
material required active participation in the learn- 
ing process and tested the student’s knowledge at each 
step of the way. The student knew immediately wheth- 
er he had answered a question right or wrong. Prac- 
tice and review were built into the course. Whether 
these advantages would carry over into the field of 
skilled craftsman training where one is faced with a 
group of men who are physically tired after a full 
day’s work and many of whom have fears concerning 
their ability to leam became the subject of a recent 
research program conducted by SRI on the problem 
of continuing education for journeymen electricians. 

We knew that teaching machines could be as effec- 
tive as teachers in communicating information but 
we didn’t know whether they would help us carry 
through an effective program of instruction for an ex- 
perienced group of adult workers. The study set out 
to answer five questions: 

1. Does the use of new teaching techniques have a 
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more positive effect on learning and achievement than 
does the more conventional means of instruction? 

2. How are journeymen’s attitudes toward training 
affected by the use of these new teaching aids? 

3. Does the effectiveness of new teaching techniques 
vary with student ability? If so, how? 

4. What are some of the effective ways of recruit- 
ing journeyman electricians to take part in voluntary 
education programs? 

5. What are some of the costs involved in the use of 
new teaching aids as compared with the cost of the 
more conventional type of instruction? 

The study began with the interviewing of a cross- 
secdon of journeymen and electrical contractors from 
San Mateo for the purpose of gaining jSrsthand knowl- 
edge of the journeyman’s on-the-job training needs. 
Following this, a survey questionnaire was mailed to all 
members of the IBEW Local 617 with the intent of 
gathering data on the journeyman’s educational back- 
ground, his attitudes toward previous training experi- 
ences, his training needs, his work aspiration and hopes 
for the future. Of the over 400 questionnaires sent out, 
65 percent were returned. 

In carrying out the experimental training program, 
ninety-six men were assigned to three modes of instruc- 
tion, each mode being made up of two classes which 
met for three hours once each week for 18 weeks. SRI 
agreed that it would only observe and evaluate the 
first nine weeks of the training, but tliat the course 
would proceed for the full semester. Of the 27 hours 
scheduled for study during the first nine weeks, five 
hours were absorbed by aptitude and pretests given on 
the first class night and a mid-term exam given on 
the ninth week of the training. Approximately 6 hours 
were given over to laboratory assignments and the 



remaining 16 hours to regular instniction (see Figure 
1). The three modes of instniction varied from indi- 
vidualized self-paced instruction on teaching machines 
to a combination of machine and live discussion for 
review purposes to conventional instruction with the 
use of advanced audiovisual aids in the third mode. All 
three groups were given the same laboratory and out- 
side reading assignments. Thus in Mode I students 
worked on self-instructional devices at their own pace. 
In Mode II, they spent most of their time on teaching 
machines, but had an opportunity at the beginning of 
the class to discuss and review with a qualified instruc- 
tor the previous week’s material and outside reading 
assignments. In Mode III they followed the more con- 
ventional live lecture-discussion method. 

The course itself was to cover the fundamentals of 
DC and AC circuit theory in one semester. This was 
done because such a review of the fundamentals of 
electricity would serve the immediate requirements of 
two types of journeymen. FIRST, those who were cap- 
able and qualified of becoming electronic specialists but 
who needed remedial training before undertaking more 
advanced study and SECOND, those who required a 
refresher course but who were not qualified or moti- 
vated to go further with electronics training. 

At the outset of the experiment there were ninety- 
six journeymen electrician students, 4 instructors, 15 
teaching machines, hundreds of feet of microfilm for 
use on the teaching machines, two classrooms and 
one womout Volkswagen for constant shuttles be-» 
tween Menlo Park and San Mateo. 

During the experiment it was found that the stu- 
dents in the teaching machine mode did only slightly 
better than those in the other two modes on the mid- 
term examination. However, those in Mode I who knew 



Figure 1 



Arrangement and Location of Classes for the Experimental Study 
of the Industrial Electronics Training Program 



Instruc- 
tional Mode 


Class 

Schedule 


Number of 
Students 


Location 


Type of Instruction by 
Allocation of Class Time 




Review 
(V 2 hr) 


Teach 
(IV 2 hr) 


Lab. 
(1 hr) 


I 


A 


15 


College 


P.I.* 


P.I. 


Idve 




B 


15 


College 


P.I. 


P.I. 


Live 


II 


D 


15 


College 


Live 


P.I. 


Live 




G 


15 


College 


Live 


P.I. 


Live 


III 


C 


18 


Union HaU 


Live 


Live 


Live 




E 


18 


Union Hall 


Live 


Live 


Live 



* P.I. = Programmed Instruction 
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more about the fundamentals of electricity before en- 
rolling in the course, as measured by a pretest of 
job knowledge, were able to perform significantly bet- 
ter on the midterm examination than did those with 
similar background in the conventional mode. Lower 
ability students also did better in the programmed in- 
structional mode than in the conventional mode al- 
though these differences were only marginally signi- 
ficant. 

Over-all, the journeymen were taught the fundamen- 
tals of a direct current theory in about half the time 
allotted to such a course. This was possible because 
of the way in which the programmed instructional ma- 
terials were developed; a thorough task analysis and 
specification of job needs has pared the course dov/n to 
its essentials. 

About two-thirds of the students in the auto-instruc- 
tional Modes I and II said it was easier to learn from 
the machine than from an instructor. The machine 
mode by itself, however, did not result in as high a 
level of over-all satisfaction with what was learned in 
the course as did the combined machine and live mode 
of instruction (Mode II) . Only about a quarter of the 
Mode I group stated they were “very satisfied” with 
their learning ejq)erience, whereas almost half of those 
in Mode II responded in this manner. Other measures 
of satisfaction with the training program indicated the 
same results. All of the students in Mode II said they 
would sign up for the same course if it were offered 
again as against 79 percent of Mode I and 87 percent 
of Mode III. Similar numbers of journeymen said they 
would recommend the course to other journeymen. 

Interest in followiag up with continued enrollment 
in a subsequent course was also much higher for the 
students in Mode II. Eighty-six percent of this group 
said that they would enroll in a course which continued 
where this one ended, as against 63 percent in the 
Mode I and 65 percent in Mode III. 

The over-all absence rate during the first half of the 
semester was five percent, excluding ten men who drop- 
ped out of the course during this period. There was 
essentially no difference in absentee or dropout rates 
between the three modes of instruction during the first 
nine weeks of instruction. 

One interesting item differentiated the enrolled jour- 
neymen most clearly from the non-enrolled group. 
The enrolled group expected to see many more changes 
in their job in the next five years. The enrolled journey- 
men who expected changes in the job, and who ex- 
pected these changes to be beneficial to ihemselves, 
were more likely to enroll in a training program than 
workers who did not feel this way. 

The enrolled journeyman tended to be less well sat- 
isfied with his current level of training. In general. 



he hoped to move up to a higher skill level than his 
current one and he valued the opportunity to do inter- 
esting and challenging work. 

The major implications of these findings seem to be 
the following: FIRST, programmed instruction worked 
as well as, or better than, conventional instruction in 
this adult education program. All of the expected ad- 
vantages of teaching machines proved true. Because 
students could proceed at their own learning rate, 
those with a higher initial knowledge of the subject 
matter moved further ahead than those in the conven- 
tional classroom. Those who were slower and showed 
less aptitude also did better in the auto-instructional 
modes. No longer did these two groups have to suffer 
because the instructor was required to gear his pre- 
sentation to the average of the group. 

While learning is still hard work, teaching machines 
take the psychological risk factor out of adult educa- 
tion. Men who have securely established themselves 
in their trade have more at stake when they go into a 
classroom — ^their future and status among their co- 
workers. They are afraid of revealing their ignorance. 
Teaching machines help overcome this fear. They gave 
the students greater confidence in their own learning 
ability. 

Teaching machines might become as common around 
the Union Hall as folding chairs and ashtrays. They 
could be used as remedial devices with journeymen 
checking them out of the Union Hall, taking a pre- 
scribed review program on their own or with occasional 
group meetings, and once having completed the course, 
checking their readiness for advanced work by passing 
a standardized achievement test. Those who qualified 
for more advanced courses would be allowed to go on. 
Slower students or those without sufficient motivation 
would be screened out. 

FINDING NUMBER 2. The combination of machine 
and live instruction yielded the highest satisfaction. 
Once the initial fascination with the gadgetry wore 
off, the human instructor resumed his importance. 
Teachers won’t be replaced by machines, but the ma- 
chines can help speed up the training job and make in- 
struction more effective and enjoyable. The teacher 
as a discussion leader helped to weld the group to- 
gether and made it a more rewarding experience than 
individual study. The machine did a better job in al- 
lowing the student to go at his own rate and to check 
himself at each step of the way. The instructor could 
draw out the students, get them to share common prob- 
lems and generally develop a feeling that they were 
a part of a meaningful training group. 

Coupling live instruction with automated instruction 
costs more but should be measured against the poten- 
tial savings in time. This experiment cut the length of 
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this course by about 50 percent. This meant that some 
of the cost saved by cutting instructional hours could 
be invested in the rental or purchase of teaching ma- 
chines and programmed materials. 

FINDING NUMBER 3. Closely related to this last 
point was the fact that commercially available pro- 
grams for presentation on teaching machines could be 
adapted to the requirements of a particular student 
group and at a cost cc nsiderably below that incurred 
by in-house programming. This experiment demon- 
strated that for about $3.50 per instructional frame 
we could obtain a satisfactory set of films for pre- 
sentation on the Auto-Tutor Machine as contrasted with 
approximately $12.00 per frame Incurred by the re- 
search team in an effort to completely reprogram one 
lesson of the course. 

FINDING NUMBER 4. Efforts to persuade journey- 
men to take part in voluntary education programs 
were substantially aided by convincing the journey- 
men that through training they could take advantage 
of the coming changes in their work. Through ques- 
tionnaires and the initial interviews the journeymen 
were given a chance to make suggestions about WHAT 
should be taught and HOW it should be taught. This 
application of the rule of participative decision making 
WORKED. The care with which the course outlines 
were prepared gave the journeyman some assurance 
that he would benefit from the training. Face to face 
discussions in the Union Hall or on the job carried on 
by the local business agent and the contractor em- 
phasized the important role they can play in getting 
journeymen to take part in training. A successful 
learning experience — one that yields satisfaction with 



training — means that employees are more likely to en- 
roll in a subsequent course than if the experience had 
not been successful. 

FINDING NUMBER 5. It was found that adult edu- 
cation required instructors who were familiar witli and 
capable of teaching adults. Adults are different than 
younger students and they must be treated differently. 
They have more at stake when they go into a class- 
room. If they fail it reflects on their work and possible 
job security. The adult applies a stricter test of the 
relevancy of the course material to his work than does 
the younger student. Like the younger student, he is 
concerned witli being accepted by his peers. He doesn’t 
want to evidence any lack of ability. He likes being 
part of a successful training group. These feelings and 
expectations can be enhanced by the proper combina- 
tion of new teaching techniques and skilled instructors. 

This study does not answer all the questions asso- 
ciated with adult education. In fact, it shows that the 
following problems must be answered. How to per- 
suade people that keeping up with technological changes 
is possible and rewarding. What is the optimum com- 
bination of live and programmed instruction? What 
is the role of other audio-visual aids such as educational 
TV, simulators, and step-by-step demonstrators? 

The study has demonstrated, however, that teaching 
machines can help answer Secretary of Labor Wirtz’s 
criticism that existing education and training institu- 
tions have not kept pace with the demands of the new 
technology. Hopefully, teaching machines offer one 
means of attracting and holding in training programs 
those who might otherwise end up among the “hard 
core” unemployed. 



TRAINING PROGRAMMERS FOR USAA 



Jean S. Moye 

Jean Moye cuHines the historical interest in and development of programmed 
instruction for use in the United Services Automobile Association. She outlines the 
manner by which USAA established an in-house capability for producing programs for 
use in training employees, and the proper Integration and management of training 
using programs. 



Interest in programmed instruction at the United 
Services Automobile Association began with the pur- 
chase of machines and programs in punctuation and 
spelling in September, 1961. Experience during the fall 
and spring with the machines and programs indicated 
that, while the material was slanted at a little lower 
level than we would have liked, training was accom- 
plished with a higher level of final proficiency in a 
shorter period. Also, less time of the company train- 
ing director, who was the instructor, was involved. 

To supplement its introductory information, the 
USAA staff continued to read all available material 
in the field. In September, 1962, the decision was made 
to embark upon a rather sizeable exploration to deter- 
mine the applicability of these techniques, not only in 
our industry, but specifically in our own organization. 
Based upon advice from many authorities in the field 
and our own knowledge of our particular needs, USAA 
decided to develop an in-house capability, and to 
employ consultants to train company programmers. 
The staff expected to gain a great deal from training 
subject-matter experts familiar with company proce- 
dure in the techniques of programmed instruction. 
USAA was fortunate in being able to take advantage 
of the availability of consultants who were in San 
Antonio. 

With our plans to enter the field of programmed in- 
struction, major decisions had to be made. First, a 
plan to canvass our employees to determine those who 
might be considered or selected as programmers had 
to be devised. It was decided to stimulate curiosity 
and motivation, so an open test was announced. A pro- 
grammed instruction exhibit was prepared in a class- 
room, and all employees were welcomed to come and 
familiarize themselves with programmed instruction. 
The exhibit was displayed for three days. Nearly all 
company employees examined this exhibit. Some em- 
ployees could see immediate uses of some of this ma- 
terial for their children. An effort was made to stim- 
ulate thinking by inviting employees to consider how 
they would feel learning the next higher level job in 
their department, or acquiring additional knowledge for 
promotion, by this method. This opportunity had not 
been perceived by the emnloyees at all. The programs 
displayed had made an impression, but they couldn’t 
perceive their usefulness at this point. 



Finally, employees interested in taking the tests for 
possible consideration for a pilot unit were invited to 
report to the cafeteria after hours on a certain date. 
More than 200, or about 15% of the employees, re- 
ported on the day of experimental testing. Ages ranged 
from 17 to 55, and positions ranged from file clerk to 
top managerial positions, representing a good spread in 
ages, background, experience, and knowledge of the 
company. The battery of different tests was given and 
scored and a composite score developed to rank, in 
order, the 200 employees. The consultants reviewed the 
personnel files and the performance records of the top 
35 on this list. Twenty-five employees were selected 
to begin intensive training. 

The day-to-day work operation of the company was 
not to be interrupted. The 25 employees were asked 
to come on a volunteer basis on Saturday mornings 
from 8 to 12 for training for a period of approximately 
8 weeks. Homework and additional study was to be 
done on their own time after working hours. 

The consultants came for the original indoctrinations 
and the inauguration of formal training. The scope 
of the material and the general instruction areas cov- 
ered included: lectures on the history of programmed 
instruction, fundamentals of programmed instruction, 
techniques of program writing, and methods of pro- 
gram evaluation. The trainees then moved into some 
practice in writing objectives, criterion test items and 
frames. Techniques of cueing, fading, and copying 
were taught and practiced. Evaluation of frames by 
pre-test and post-test scores and error count was taught. 

The first evaluation of the 25 trainees was made to 
determine those who had the most promise of becom- 
ing effective program writers. After six weeks, the 
first cut-back was made, reducing the group to 17. Dur- 
ing the next six weeks, 7 trainees withdrew for various 
reasons. The program-writer trainees were put in a 
study-work application situation from almost the begin- 
ning. By the third week, they were beginning to pro- 
duce some programmed materials. The quantity of the 
work produced by these people was so great that the 
editing and review of this material consumed a great 
deal of the consultants’ time. At the same time, qual- 
ity was beginning to appear. The remaining group, 
consisting of 9 programmers and one editor, had pro- 
duced 7 usable programs, and had 16 other programs in 
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progress. At this time, the group had to be reduced to 
6 programmers, due to budget limitations, with three 
trainees returning to their previous jobs but rem ainin g 
on stand-by. 

During this initial period, many peripheral training 
areas were explored, in order to increase the under- 
standing of programmed instmction within our organi- 
zation. Consultants and program representatives were 
concerned not to embark on projects so esoteric that 
the line organization and top management of the com- 
pany would be unfamiliar with this technique and there- 
by possibly offer some resistance or hesitancy in either 
allowing programmed material in their areas, or by not 
cooperating or proffering information and technical 
help. 

To insure proper use of programmed instruction, it 
was felt that outside training for supervisory and 
management personnel was essential. Staff people 
were encouraged to enroll at the two local universities, 
at which the consultants were teaching classes in 
methods of programmed instruction. In this way, they 
were not embarrassed, nor was their prestige in any 
''•'ay affected when they were included in general orien- 
tation classes with the program writers themselves. 
These staff people were assigned to the two universities, 
with the higher level people in one course and the line 
supervisors and the department heads in a course at 
the other university. 

A concentrated effort was also initiated to establish 
a current library in the field of programmed instruc- 
tion. Some of these materials were placed in the em- 
ployee library, and the remainder in the training di- 
rector’s office. The specific technical tr ainin g for the 
programmers, and the general orientation for the su- 
pervisory staff and managerial people, were the com- 
ponent parts of an overall two-pronged approach to 
properly launching programmed instruction at USAA. 
This approach would be assured of understanding and 
support on the part of top management and the line 
supervisory staff, while ensuring that the program wri- 
ters would be proficient and competent to carry off 
their part of the project. 

After the programming unit was organized, a shake- 
down period brought out some rather interesting points. 
First of aH, several transitions had to be made. Train- 
ees had been selected for their role of program writer 
on the basis of their background, experience and their 
subject matter knowledge of the operation of the com- 
pany. They were to use their subject matter knov/1- 
edge to avoid repeated interruption, annoyance and 
dependance on the line people, the real technical ex- 
perts, for every piece of information. Program writ- 
ers were told to make use of the subject-matter mate- 
rial acquired during their time with our organization. 



Jean S. Moye 

but they were no longer to be the subject-matter ex- 
perts per se. They were to be experts in the tech- 
niques of program writing, but their material was to 
be referred to technical experts for validation and 
technical correctness of the material to be taught. On 
the other hand, these staff personnel and technical ex- 
perts were not to encroach upon a writer’s style or 
technique in writing the program. This resulted in a 
rather significant shift of emphasis for the program- 
mers. 

Another very revealing experience was to observe 
the reaction of these people who had come from highly 
organized, highly oriented interdependent production 
units, and who were now stepping across the line into 
an unfamiliar area where they were completely inde- 
pendent producers. A great deal of responsibility was 
placed upon these people to be creative and imaginative 
in a way which they had not experienced before. They 
had some difficulty in giving up the security and safety 
of a highly organized, structured work environment, so 
a deliberate effort was made to break up old patterns. 
They had no rigid lunch schedule, no timed coffee 
break. By intentionally dis-orienting an organized pat- 
tern, it was felt that these programmers were given 
an important assist in stepping over the line into the 
area of independent thinking, and acceptance of re- 
sponsibility for a project. 

At this time, the unit was comprised of seven women. 
Two young men included in the college program, who 
had been through the training program, were on stand- 
by until budget or other indications permitted an in- 
crease in the staff. There were problems of temper- 
ament and personality. The plans were to develop pro- 
grammers as individuals and to give them full rein 
to discover their capabilities. The restraints of rigid 
schedules have been removed. This was dis-orienting 
for a time, but natural team relationships are develop- 
ing. One programmer worked with another program- 
mer on one project and then the team changed to an- 
other project. This has proved effective in providing 
scope, depth and versatility. This flexibility allows a 
combination of talents to complement themselves to 
produce the kind of programs USAA needs. 

After only six months, members of top management 
became very pleased at the promise that programmed 
instruction held. They found that it had applications 
in new areas. They also found that some possible 
applications of programs are not significantly more 
effective or less expensive than conventional training 
methods. No programs will be developed in these areas. 

One of the most important by-products of this project 
has been the effect on our supervisory force in their 
more critical examination of some of the operations 
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in which they have been involved, and a fuller recogni- 
tion of what some of their training problems are and 
have been. Now, there is a very keen desire on the 
part of the supervisory staff to discuss how they can 
help get those employees trained in a shorter period of 
time. 

Our present thinking and plans regarding the im- 
plementation and administration of completed pro- 
grams call for classroom presentation of programmed 
instruction packages, under the control of the training 
director and the supervisor of the programmed in- 
struction unit 

Future plans, reached after deliberation of the many 
factors involved, call for the following: (1) For the 
next one to two years, it is essential that all possible 
feed-back fiom student employees be obtained, so that 
programs can be modified if necessary. (2) Line su- 
pervisors ought to be relieved of as much of the training 
load as possible. Not every supervisor is a trained 
instructor, and would need, in addition, special train- 
ing to administer program packages. (3) New em- 
ployees, after the usual orientation on the first day of 



employment, can be easily put in a classroom situation 
to work through programs developed for the particular 
department and position for which they were hired. 
When reporting to the unit supervisor, new employees 
are then prepared, to whatever degree we have pro- 
grammed, to begin the new job better equipped to per- 
form productively in a shorter period of time. (4) Can- 
didates for promotion, after selection by the supervi- 
sor, will be referred back to the training director, for 
administration of whichever programs are considered 
applicable for the particular upgraded job. 

USAA feels the above outlined program will greatly 
enhance the understanding of the role and effectiveness 
of the training director, the program writer and the 
line supervisor, and, in addition, will hasten the time 
until the program becomes operational and of practical 
value to the company. 

This company is moving rapidly in the production 
and use of programs for training. It hopes to be in the 
forefront with those who are producing valuable, worth- 
while, competent materials within a short period of 
time for the whole industry. 
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APPLICATION OF AUDIO<LINGUAL PROGRAMMING TO INDUSTRIAL TRAINING 

Robert Brink 

An audio-lingual program was developed for use in training operators to give tele- 
phone numbers to inquirers. The programmed course achieved a 40% reduction in 
training time, 52% increase in passing scores and an 85% increase in perfect scores. 



Before I get into the body of my remarks let me say 
that I am not an experimental psychologist. I have had 
neither training nor experience in that field. It is not 
my desire, therefore, to take sides with respect to the 
conflicting theories. My purpose, rather, is to share 
witri you some of the experiences that we had in devel- 
oping a course for training information operators in the 
General Telephone Company of California. 

Now the popular concept is that an information op- 
erator’s job is quite simple. She is required only to 
look in the directory and report the number if she 
finds it. If she does not, she so informs the customer. 
However, in the metropolitan Los Angeles area an op- 
erator has six directories containing over 2 million 
listings to which she may have recourse in searching 
for any particular listing. Furtliermore, she has a 
daily supplement for each directory showing new list- 
ings and changed listings. 

In addition, as you can understand, customers rarely 
ask for listings precisely as they are worded in the rec- 
ords, so there is necessarily a great body of practice 
which our operators must be taught with respect to 
search procedures, questioning techniques and report- 
ing. In this regard, it might interest you to know that 
in the Los Angeles Central director;' there are some 
3,500 listings for the name Johnson alone. Yet our ob- 
jective is to provide a report acceptable to the cus- 
tomer within less than 40 seconds, on the average, 
from the receipt of his order. In fact, 20 to 30% of 
the numbers are quoted by the operators within 15 
seconds. This emphasis on speed may seem excessive. 
However, let me remind you that information service 
is a non-revenue-producing service. Its provision costs 
over $2,000,000 per year in our California company 
alone. So, taking the 40 second average disposition in- 
terval as a guide you can see that any training ap- 
proach which could result ia reduction of this average 
by only one second would produce a savings of $50,000 
annually. Furthermore, the grade of service accept- 
able to our subscribers and to the Public Utilities Com- 
mission of California, will not tolerate an error factor 
higher than 1.7%. Since an operator will handle in 
excess of 600 calls per day in a busy office she cannot 
make errors of any type on more than 10 calls if she 
is to achieve the accuracy factor we demand. There- 
fore, I think you can realize that we are caught be- 
tween a Scylla and Charybidis of providing a high 



grade of service on one hand and doing so economically 
and extremely quickly on the other. So the complexities 
of information services are far greater than might ap- 
pear at first blush. 

The question became then, how could the learning or 
training situation itself be manipulated to produce the 
maximum transfer to the job situation? It is generally 
accepted that in any specific application the transfer is 
improved by increasing the amount of similarity be- 
tween the training situation and the job situation. In 
our situation we were faced with the fact that our prin- 
cipal requirement is for vocal contact between an op- 
erator and customer. After some consideration and 
study we felt that the use of tape recorded instruction 
was more nearly reahstic to our job situation than an 
effort to do the same programming in written form. 

In the design of our course, we attempted to estab- 
lish “task set” by frequent use of such phrases as “sup- 
pose a customer were to make the following request,” 
thereby giving meaning to each problem and establish- 
ing a reahstic matrix within which to consider its 
solution. We made hberal use of what I call “walk 
through” calls during which we took a student step by 
step through the procedures necessary to arrive at a 
proper report. The characteristics of these calls have 
been very carefully designed to help a student estab- 
lish criteria by which to judge her subsequent deci- 
sions. Then we provide extensive practice caUs with 
the complexity of the recall required quite low for 
each reinforcement. We attempted throughout to 
provide an optimal schedule for the seeding of review 
material with the result that there are over 600 prac- 
tice examples found in the course. 

In addition to the lesson tapes themselves, we use 
other teaching aids including instructor manuals cov- 
ering equipment operation and training room adminis- 
tration; student work books to provide drill material 
and listing examples for purposes of comparison; peri- 
odic written quizzes, in multiple-choice form, as a 
check on learning progress; and a classroom analysis 
form on which to record student progress and comple- 
tion schedule. 

The results that we have achieved with the program 
have been quite satisfying. In the first place, we 
have been able to reduce the length of our initial train- 
ing, and therefore its cost, by 40% . We had used ten 
days for information training prior to this time, and 
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as a result of this course have reduced the training 
time to six days. 

We have now trained 237 girls on the audio-lingual 
program. One hundred and two of them have com- 
pleted their three-month probation period. The aver- 
age personal index in the third month has been 91.2%, 
we’re looking for 85% or better at this point. But more 
rewarding we have had a 52% increase in passing 
scores and an 85% increase in perfect scores. 

Meanwhile, the quality of service in the oflBces has 
greatly improved as students trained under the new 
method have been absorbed into the force, and cost of 
providing service has declined. So we feel the program 
is eminently successful. 

We have realized several ancillary benefits from the 
new course. One has been the standardization of our 
training, not only among students in one office but be- 
tween offices throughout our company. This has ad- 
vantages which are obvious to all of you, principally 
as regards subsequent on the job supervision and inter- 
office transfers. 

Second, and this is not necessarily intrinsic to the 
tape recorded type of progr ammin g used, we have a 
ready library of refresher training material which can 
be given as need arises to experienced operators. Such 
refresher training or training of re-hires and girls with 
previous experience can be done concurrently with a 
new training class, since aU students work inde- 
pendently, and without requiring additional ir .tractor 



time. This has provided an additional savings over 
and above the initial 40% reduction in training costs. 

Third, the instructor, being freed of much of her pre- 
vious responsibility for teaching routine material, is 
able to give closer attention to each student’s develop- 
ment and is also able to overlap other duties while 
supervising the trJMuing room. 

Fmally, through standardizatiim of our operator 
training and establishment of mor6 T^'alistic measures 
of terminal behavior, I am ho>^ing tl\iit we shall be 
able to contribute to a defin^tii^s t-tady of applicant 
testing with benefit to recruitment 9,nd labor turnover. 

Let me end my remarks by saying that whatever 
programmmg skills we have acquired are not most effi- 
ciently transmitted through conference methods. This 
is particularly true since the success of a program is 
dependent upon very fine discrimination in technique 
which are by no means apparent from a study of the 
completed product. These have been arrived at largely 
through trial and error. 

Like an indulgent parent trying to direct an unso- 
phisticated child, I should like to be able to guide your 
path to avoid some of the pitfalls through which our 
experience has led us. I am convinced, however, that 
contention with these problems is in itself salutary. 
Unfortunately, programming is still more an art than 
a science. As such, it must be nurtured and devel- 
oped through experimentation and practice. 
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A TECHNIQUE FOR DETERMINING THE ADMINISTRATIVE FEASIBILITY OF 
PROGRAMMED LEARNING IN GOVERNMENT AND INDUSTRIAL TRAINING PROGRAMS 

James E. Gilbert 

James E. Gilbert presents a technique for quantifying the feasibility of programming 
training tasks. The quantification is based on a ranking of each course against four 
criteria of each administrative feasibility. The rankings are converted to Program 
Feasibility Scores, which can be categorized and compared either between or within 
subject matter areas. 



In order that any instructional method may be most 
economically and eflSciently employed, it must be 
determined which training programs would be most 
suited for its application. Where will the use of pro- 
grammed learning techniques bring the greatest return 
for the time, manpower, and money involved? In 
other words, in what courses sponsored by an industri- 
al or government training organization would it be the 
most feasible to use programmed learning? 

The subject of feasibility is concerned with two areas: 
administrative feasibility and technical feasibility. 
Administrative feasibility is defined here in terms of 
training requirements, difficulty of meeting such re- 
quirements, stability of subject matter, and availability 
of individuals knowledgeable in the subject matter and 
in programmed learning techniques in order to pre- 
pare adequate auto-instructional materials. Technical 
feasibility, on the other hand, is defined in terms of 
the training of programmers, programming methods 
appropriate to the subject matter, display formats, 
and facilities to evaluate programs. It is the plan of 
this paper to concern itself with administrative feasi- 
bility, leaving technical feasibility (with its myriad 
problems) for another time. 

The specific problem of this paper was to develop a 
technique which would indicate those courses spon- 
sored by a government or industrial training organi- 
zation in which it would be administratively feasible 
to use programmed learning methods. This technique 
was concerned with both complete and partial pro- 
gramming of courses, in that the adniinistrative critera 
“hold” regardless of the extent of the course pro- 
grammed. The question of partial or complete pro- 
gramming more appropriately belongs to the area of 
technical feasibility anyway. 

Operationally, (Gilbert, 1961) administrative feasi- 
bility was defined in terms of our criteria a course 
must meet to be considered administratively feasible 
to program. (1) Requirement — the number of trainees 
requiring a particular course. (2) Difficulty — ^the de- 
gree of difficulty in meeting the training requirement 
now or in the near future due to staffing, fiscal, 
space, or instructional problems. (3) Stability — ^the de- 
gree of subject matter stability in terms of amount and 



frequency of revision. (4) Technical Assistance — avail- 
ability of technical personnel to prepare auto-instrac- 
tional programs. In general, the technique consists 
of the following steps. Courses are first grouped by 
subject matter. It is necessary to have at le^st five 
courses per subject matter grouping. If this is not 
possible, a miscellaneous subject matter area must he 
created and courses placed in it until an N of at least 
five is obtained. Responsible training officers then 
judge each course on the basis of the defined adminis- 
trative criteria and rank tliese courses on each of the 
four criteria. These ranks are converted to a criterion 
score by comparison with a nomograph (Table 2). 
The criterion score is then summed across criteria 
and a Program Feasibility Score obtained. Based on 
the Program Feasibility Score, a Program Feasibility 
Category is assigned. The Program Feasibility Cate- 
gory indicates the degree to which a course is admin- 
istratively feasible. The Program Feasibility Score 
provides guidance to the training officer in comparing 
the administrative feasibility of courses. Detailed in- 
structions for this procedure may be found in Table I. 

Whenever possible, multiple rankings of courses on 
the four criteria should be obtained. When the judg- 
ments of more than one training officer are available, 
a Program Feasibility Score is determined for each 
course, for all judges. These Program Feasibility 
Scores are then averaged. The average Program Feas- 
ibility Score for each course is then treated as if it were 
a PF Score based on only one judgment and assigned 
a Program Feasibility Category. Obviously, scores 
based on the judgments of more than one knowledge- 
able training officer are more reliable. 

For purposes of example, consider two subject mat- 
ter groupings: (1) mathematics and (2) office practices. 
The courses have already been grouped by subject 
matter. These courses will be found listed on a Pro- 
gram Feasibility Worksheet (Figure 1) by subject 
matter. In the first four columns to the right of the 
course title appear the criteria ranks of the courses 
on the four criteria. “Introduction to Modern Alge- 
bra I” for example, ranks sixth on requirement, four- 
teenth on difficulty, tenth on stability, and ninth on 
technical assistance. “Key Punch Operation”, on the 
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Other hand, ranks first on requirement, fourth on dif- 
ficulty and stability, and third on technical assistance. 
Continuing to the right, consider the next five colunms. 
These are the criteria scores. The ranks have been 
converted into criteria scores by using Table 2. Let 
us find the criteria score for “Introduction to Modern 
Algebra I”, requirement rank 6. Table 2 is entered 
veitically by finding the total number of courses ranked 
(16) and horizontally by locating the rank of that 
course on the specific criteria (6). Where the low (6) 
and the column (16) intersect the criteria score (66) 
will be found. This process is followed for all courses 
in both subject matter areas. When criteria scores 
are entered for all courses the sum of these criteria 
scores is computed by adding the criteria scores for 
the four criteria for each course. This is the Program 
Feasibility Score (PF Score). The PF Soore for “In 
troduction to Modem Algebra I” is 170, while for “Key 
Punch Operation” is 200. 

After the PF Scores have been computed, they are 
compared with the right-hand column of Table 3 and 



the proper Program Feasibility Category (PI _dte- 
gory) is assigned by checking one of the five appro- 
priate columns to the right. By counting the checks in 
each column (1 through 5) the number of courses in 
each PF Category may be determined. Table 4 indi- 
cates the frequency of courses in each category by sub- 
ject matter (mathematics, office practices). Inspecting 
the results of the application of this technique to the 
two subject matter areas in question, mathematics 
seems the most probable to consider programming, 
at least administratively, and within that subject mat- 
ter area the following courses are the most adminis- 
tratively feasible: “Elementary Algebra”, “Calculus 
I”, “Caculus II”. This example has been presented as 
the result of the judgment of one training officer. It is 
hoped that, in the majority of applications of this tech- 
nique, the judgments of a number of training officers 
will be used. If multiple judgments had been used, the 
PF Scores for each course would have been averaged 
and this average PF Score treated in the same manner 
as a single training officer PF Score. 



Table 1 

Instructions for Determining the Administrative 
Feasibility of Programmed Learning 



Slep 



Instruction 



Outcome 



1 . 

2 . 

3. 

4. 

5. 



6 . 

7. 

8 . 



Group courses into subject matter areas (must be at least five courses 
in each subject matter area) . 

Make appropriate entries on the Program Feasibility Score Worksheet 
under “Subject Matter Area” and “Course Title” columns. 

Rank each course in the subject matter area on the four administra- CRITERIA RANKS 
five criteria to obtain criteria ranks.* 



Convert the criteria ranks to criteria scores using the Criteria Rank 
Conversion Table (Table 2) . 

Sum the criteria scores for each course to obtain the program feasi- 
bility score (PF Score). (Where multiple judges are available, 
average the PF Score for all judges and continue with step 6 using 
this mean score.) 

Compare the PF Score with the PF Category Score Bounds Table 
(Table 3) to assign each course a program feasibility category 
(PF Category). 

Indicate the PF Category for each course by placing a check in the 
appropriate box on the worksheet. 

Continue this process until all courses in all areas have a PF Score 
and PF Category. Compare PF Category frequencies across areas 
and PF Scores across courses to determine the courses and areas 
most feasible to program. 



CRITERIA SCORES 

PROGRAM FEASIBILITY 
SCORE 



PROGRAM FEASIBILITY 
CATEGORY 



* In assigning ranks use the numeral 1 to signify that course with the largest training requirement, that course with the greatest 
difficulty, etc. As the courses decrease in training requirement, difficulty, etc., the ranks should be of greater magnitude. Tied 
ranks are not permitted. A decision must be made and all courses must have a different rank on each criterion. 
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The rationale concerning the technique proposed, 
some pertinent remarks concerning the steps in the 
application of the technique, and the derivation of some 
of the tools used by the technique will now be dis- 
cussed. The criteria of administrative feasibility were 
limited to the four defined, as those are the most spe- 
cific to both programmed instruction and government or 
industrial training. Two notable exceptions to the 
criteria are money and time. These were not included 
as some fiscal considerations are subsummed under 
difficulty and technical assistance while others are 
taken care of by budgeting for conventional training 
courses normally administered. Time factors, although 
important, were considered as a constant as time nor- 
mally is allocated to course preparation regardless of 
the method of teaching employed. In the judgment 
of training officers and psychologists, the four criteria 
stated are the most relevant to the concept of admin- 
istrative feasibility. 

Certain remarks are pertinent concerning the steps 
followed in determining the programming feasibility 



of courses. When the courses are arranged in subject 
matter areas, five is a required number, as this is the 
smallest group that can be assigned criteria scores 
meaningfully. Ranking courses on the four criteria 
in groups less than five tends to be too artificial and 
assigning scores from a nomograph to such a small 
number results in criteria scores that are spuriously 
high. As stated earlier, courses that cannot be placed 
in a subject matter category can be grouped together in- 
to their own category and judgments made in terms of 
this miscellaneous group. When ranking the courses 
on each of the four criteria, all data available to the 
training officer should be used. Planning documents, 
budget programs, manpower estimates, trainee re- 
quirements, in terms of skills and numbers, should all 
be considered. The more objective data available upon 
which to assign ranks to different courses across cri- 
teria, the more meaningful will be the resulting Pro- 
gram Feasibility Score. 

Converting the ranks to criteria scores was included 
in order to be able to compare course program feasibility 



PROGRAM FEASIBILITY WORKSHEET 


Subject 


Course Title 


Criteria Ranks 


Criteria Scores 


PF 


PF Category 




Matter Area 


Reqr, 


DW. 


Stab. 


Tech. 


Reqr. 


DijSf. Stab. 


Tech. 


Score 1 


2 


3 


4 


5 


Mathematics 


Elementary Algebra 


1 


1 


1 


15 


97 


97 


97 


9 


300 


V 










Intermediate Algebra 


15 


2 


2 


14 


9 


91 


91 


16 


207 




V 








Fundamentals of Math. 


8 


9 


5 


16 


53 


47 


72 


3 


175 




V 








Ele. of Math. Anal. I 


3 


5 


12 


7 


84 


72 


28 


59 


243 




V 








Ele. of Math. Anal. II 


4 


6 


13 


8 


78 


66 


22 


53 


219 




V 








Calculus I 


7 


10 


3 


1 


59 


41 


84 


97 


281 


V 










Calculus li 


9 


11 


4 


2 


47 


34 


78 


91 


250 


V 










Intro, to Modem Alg. I 


6 


14 


10 


9 


66 


16 


41 


47 


170 




V 








Intro, to Modem Alg. II 


13 


15 


11 


10 


22 


9 


34 


41 


106 






V 








5 


12 


7 


4 


72 


28 


59 


78 


237 




V 




















22 


53 


72 


163 






V 




t 


Matrix Theory 


2 


4 


1 
























Intro, to Complex Variable 


11 


8 


15 


12 


22 


35 


















Vector Analysis 


12 


7 


16 


13 


28 


59 


3 


22 


112 






V 




Office Practices 


Basic Typing 


4 


1 


1 


2 


30 


90 


90 


70 


280 


V 










Advanced Typing 


2 


2 


2 


1 


70 


70 


70 


90 


300 


V 










Key Punch Operation 


1 


4 


4 


3 


90 


30 


30 


50 


200 




V 








Office Procedures 


3 


3 


5 


4 


50 


50 


10 


30 


140 






V 






Refresher Shorthand 


5 


5 


3 


5 


10 


10 


50 


10 


80 








V 



Figure I. Mathematics and Office Practices Program Feasibility Worksheet 
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across subject matter areas. Converting to criteria 
scores also l\as advantages in treating the data, in- 
terpreting tlu data meaningfully, and putting the data 
into a form that is more appropriate for empirical 
study. It was important to consider the type of studies 
that will be carried out in the future and apply a tech- 
nique that could easily be adopted to future needs. 



Rankings were converted to criteria scores by using 
the formula (Ward, W. H. 1961) : 

100 R - 50 

Criteria Score = 100 — 

N 

Where: R = Rank assigned to a course. 

N = Subject matter area size. 



Table 2 

Criteria Rank Conversion Table 

This table converts rankings to a criterion score scale with a mean of 50. To use the table, first determine the 
number of courses ranked. Then enter the table with the rank of the individual course on the criterion (a rank 
of 3 indicates a course that is third from the top). At the intemection of the row and column will be found the 
criterion score. 



Number of Courses Ranked 



Rank 


5 


6 


7 


5 


P 


10 


77 


72 


IS 


14 


75 


16 


77 


18 


19 


20 


Rank 


1 


90 


92 


93 


94 


94 


95 


95 


96 


96 


96 


97 


97 


97 


97 


97 


98 


1 


2 


70 


75 


79 


81 


83 


85 


86 


88 


88 


89 


90 


91 


91 


92 


92 


92 


2 


3 


50 


58 


64 


69 


72 


75 


77 


79 


81 


82 


83 


84 


85 


86 


87 


88 


3 


4 


30 


42 


50 


56 


61 


65 


68 


71 


73 


75 


77 


78 


79 


81 


82 


82 


4 


5 


10 


25 


36 


44 


50 


55 


59 


62 


65 


68 


70 


72 


74 


75 


76 


78 


5 


6 




08 


21 


31 


39 


45 


50 


54 


58 


61 


63 


66 


68 


69 


71 


72 


6 


7 






07 


19 


28 


35 


41 


46 


50 


54 


57 


59 


62 


64 


66 


68 


7 


8 








06 


17 


25 


32 


38 


42 


46 


50 


53 


56 


58 


61 


62 


8 


9 










06 


15 


23 


29 


35 


39 


43 


47 


50 


53 


55 


58 


9 


10 












05 


14 


21 


27 


32 


37 


41 


44 


47 


50 


52 


10 


11 














05 


12 


19 


25 


30 


34 


38 


42 


45 


48 


11 


12 
















04 


12 


18 


23 


28 


32 


36 


39 


42 


12 


13 


















04 


11 


17 


22 


26 


31 


34 


38 


13 


14 




















04 


10 


16 


21 


25 


29 


32 


14 


15 






















03 


09 


15 


19 


24 


28 


15 


16 
























03 


09 


14 


18 


22 


16 


17 


























03 


08 


13 


18 


17 


18 




























03 


08 


12 


18 


19 






























03 


08 


19 



20 02 20 



Score Bounds of PF Categories 



PF 

PF Category Percentages Score Bounds 



1 


Definitely program 


15 


313 or above 


2 


Probably program 


20 


249 to 312 


3 


Question 


30 


153 to 248 


4 


Do not program 


20 


89 to 152 


5 


Definitely do not 
program 


15 


88 or below 



Frequency of Mathematics and Office Practices 
Courses in PF Categories 



Mathe- Office 

PF Score PF Category matics Practice 



313 or above 
249 to 312 
153 to 248 
89 to 152 
88 or below 



1 Definitely program 

2 Probably program 

3 Question 

4 Do not program 

5 Definitely do not 

program 



0 

3 

6 

3 

0 



0 

2 

1 

1 

1 



Table 3 



Table 4 
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The formula produces scores ranging from 0 to 100 
in a rectangular distribution with a mean of 50. The 
standard deviation varies with N due to the rectang- 
ular distribution. A rectangular distribution rathei th^a 
a normal distribution was used due to the fact that 
with ranks, an equal number of cases falls within each 
category, each of the four criteria must have tire same 
number of cases at each rank. Table 1 is the result 
of carrying out the computation of the formula for 
groups of courses from five to twenty. 

In establishing the Program Feasiiiility Category, the 
percentage distribution in Table 3 was used for the 
PF Score for each of the five categories; these per- 
centages were also used to establish the score bounds 
for each of the five categories. The lowest PF Score 
that can be obtained by any course is 40, where the 
size of the subject matter area is five and the rank is 
five on aU four criteria. The highest score obtained for 
this same subject matter area is 360, a rank of 1 on 
all four criteria. The total score range, therefore, is 
320 score points (360-40) and this with a mean of 200 
(4x50). Multiplying this score range by the percent- 
ages the score bounds reported on Table 3 are obtained. 

At the present time, the percentages used (15, 20, 
30, 20, 15) represent only best judgments based upon 
limited observations. They must await empirical 



validation studies before they can be used as anything 
more than approximate guidelines. If different per- 
centages are desired, either more of less rigid, they 
may be substituted with minimum difficulty, requiring 
oniy li change In score bounds. With these considera- 
tions in mind, only a course in the upper 15% could 
be very seriously considered as administratively feas- 
ible to program (PF Category 1). Those courses falling 
in the next 20% (PF Category 2) could be challenged 
while tliose in the middle 30% (PF Category 3) could 
be very seriously questioned if considered for program- 
ming at all. Courses in the lower 35% of those judged 
were so low as to preclude consideration (PF Cate- 
gories 4 and 5). 

After identifying the most feasible subject matter 
area the training officer may then select within that 
area the courses of greatest promise. Knowing which 
courses are of high administrative feasibility, the train- 
ing officer is in a much better position to make judg- 
ments concerning what may now be properly called 
technical feasibility. The government or industrial 
training organization is also in a position that will allow 
it to plan more adequately for effective, efficient and 
economical application of programmed learning meth- 
ods to training functions. 



DETERMINING TRAINING NEEDS 

Joseph M. Madden, Major, USAF 

Joseph Madden discusses the need for detailed specifications of training needs and 
course objectives, not only for programmed instruction, but for all other types of 
instruction as well. He also discusses a combination procedure for developing a 
"behavioral task statement," based on detailed task-analysis and serving as a solid 
basis for the development of training materials. 



The objective of a training program may be stated 
as teaching people to respond in a certain way in a 
given situation, that is, to perform certain required 
work at the time it is needed. In order to attain this 
objective, a cycle is followed which delineates four 
large divisions of activity. These divisions are: de- 
termination of training needs, design of training cur- 
ricula congruent with these needs, presentation of the 
training to people, evaluation of the training given in 
terms of the curricular objectives, and, to begin the 
cycle again; determination of additional training needs. 

All phases of the cycle are important. The effects 
of improperly accomplishing any one phase results in 
failure of the entire training effort. The man is not 
adequately prepared for what he has to do, he has been 
overprepared, and money has been wasted in giving 
him training which he does not use, or the liming is 
off and he is trained too long before the training is 
needed or not soon enough. Following this line of 
thinking, it quickly becomes apparent that the salient 
focus for most efficient and economical training should 
be on that phase of the cycle concerned with determin- 
ing training needs. An overstatement of needs results 
directly in excessive costs. An understatement of 
needs produces results similar to erroneous practices 
in the other three cycles so that man cannot do the re- 
quired work effectively. 

Not only does an emphasis on cost control make the 
determination of training needs extremely important, 
but this part of the four-phase cycle has historically 
been a course of considerable difficulty concerning 
which very little progress has been made. We have 
learned much about curriculum design and the training 
itself, less about training evaluations and still less about 
the technology of accurately determining training needs. 
Since we are dealing with a four-phase cycle, the suc- 
cess of which depends upon efficient accomplishment 
of each phase, it seems appropriate to direct attention 
toward the weakest of the four. 

The determination of training needs may be broken 
down into two distinctly different types of activity. The 
first is describing the work activities to be done by the 
individual to be trained, and the second is inferring the 
training needs from this description. 

The description of the work to be done forms the 
keystone not only of the training program, but every 



phase of personnel management. Every aspect of the 
training program is directly dependent on the accuracy 
and comprehensiveness of the work description. In 
spite of this, describing work, usually in terns of units 
called jobs, has been treated generally in a rather casual 
manner. A paragraph or two of descriptive material is 
even considered as adequate coverage for an entire 
occupation in some circles. Inference of training needs 
from such descriptions cannot approach a reasonable 
degree of accuracy. More detailed and comprehensive 
information is needed and the information must be in a 
form which facilitates inference of training needs. 

I would like to describe a method used in the Air 
Force which has been developed to provide the quality 
and detail of work activity information needed for in- 
ferring training needs as well as several other purposes. 
Although this method is presently unique in the Air 
Force, considerable interest has been shown in it by the 
other military services, governmental agencies, and 
industrial concerns. Furthermore, the method is general 
in that it can be applied in any situation where work 
descriptions are obtained. 

The Air Force method utilizes a task inventory to 
survey the work of incumbents in jobs, specialties, 
career-leaders, or general occupation areas. The task 
inventory consists of a listing of pertinent task state- 
ments organized into categories called duties. Each task 
statement describes a bit of behavior and an effort is 
made to keep these bits about equal in size, that is, at 
the same level of specificity. An attempt is also made 
to make each task statement descriptive of behavior 
which is independent of that expressed by other task 
statements. A prime feature of the method is that the 
incumbent not only checks off those tasks which he 
performs, but he writes in additional tasks which he 
does but which are not listed. In addition, he is re- 
quested to rate all tasks which he performs on one or 
sometimes two rating scales. The dimensions of work 
covered by the rating scales depend upon the purpose 
of the survey. Ratings may be concerned with such 
topics as the relative proportion of time spent on each 
task, where the task was learned such as on the job or 
in a technical school, the proficiency level on each task, 
and so on. 

Administration of the inventories usually takes place 
in a testing room, one of which is located on practically 
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every Air Force Base. The booklets are mailed to 
testing officers with a list of the number and types of 
incumbents to whom they are to be administered. The 
testing officer obtains the desired incumbents through 
the appropriate commander, administers the inventories, 
and returns them to the Personnel Research Laboratory 
for analysis. 

The method I have very briefly described is consid- 
ered a combination of the best features of traditional job 
analysis procedures. For instance, the open-ended fea- 
ture of the questionnaire method is retained by provid- 
ing the opportunity for writing in task statements, and 
interviews and observation of work being performed 
may occur during construction of the inventory. The 
task inventory method is simple, provides work infor- 
mation in a standard, qualified form accurately and 
comprehensively, provides current information, permits 
inexpensive broad sampling, serves many using agencies, 
and is flexible in terms of sample selection. A good 
deal of research has been completed and reported in the 
course of development of the task inventory method 
and much of the rationale is supported by factual 
findings. 

The end result of a survey utilizing the task inventory 
method is a list of tasks performed by incumbents in 
the occupational unit surveyed. Each of these tasks is 
described in terms of a number of variables, depending 
on the purpose of the survey and the rating scales used. 
K the determination of training needs has been one of 
the objectives of the survey, various kinds of pertinent 
data would be available. These would include such 
items as the proportion of incumbents who perform 
each task, whether the task was learned on the job or 
in the schoolroom, whether or not incumbents feel a 
need for more or better training for each task, and so 
on. We thus have a very complete behavioral descrip- 
tion of what the training program is intended to prepare 
people to do, barring abrupt changes in the work con- 
tent. We have avoided vague and often meaningless 
statements such as, “must have knowledge of . . .” or 
“must be able to . . . .” Our task statements are clear 
and concise. We are now ready to approach the task 
of making inferences concerning the training needed to 
accomplish the work tasks we have listed. 

With regard to the inference of training needs, it 
should first be pointed out that the nature of the be- 
havioral task statement reduces or may even eliminate 
much of the difficulty involved in determining training 
needs. For instance, if the task statement is, “Remove 
and replace C-47 engine,” the training objective is not 
difficult to state. Requirements for knowledge of theory 
or principles, of course, are dependent upon inferences 
that are not simple, but this is true in any system. If 
the scope of work covered is narrow, the task of infer- 
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ring training needs is much simplified. That is, if the 
task inventory covers only a single job such as, pastry 
cook, or battery maintenance man, a large number of 
complications are avoided which must be dealt with 
when larger work units are used. In most cases, how- 
ever, a larger unit such as a whole occupation, or spe- 
cialty as it is called in the Air Force, is the subject of 
concern. Specialties contain a varying number of jobs 
at all levels of skill, from helper to superintendent. The 
specialty is the unit of Air Force management and prac- 
tically all personnel actions are based on it. It corre- 
sponds to what is often called a job in industry. Let us 
list just a few of the more dominant problems resulting 
from the fact that a rather large work unit is employed. 

The first problem, which concerns the specialization- 
generalization controversy, is not unique to the Air 
Force. It seems to be of general concern, even for 
small organizations. We usually settle this one in a 
vague and tenuous way. We say something about 
“broad job concepts” and end up attempting to prepare 
the individual to perform all the work in a specialty at 
the apprentice, journeyman, or supervisory skill level 
and often at all three skill levels. We are then imme- 
diately faced with another general problem, “What is to 
be learned in the classroom and what on the job?” 
Most of us have some sort of of an approach to this 
question which draws a line between theory, principles, 
and general information on the one hand, and skills, 
practice, application, and implementation on the other. 
The third problem in this general area is the temporal 
application of training, determining the experience 
points at which to provide for further, higher level 
training. We could spend a great deal of time on each 
one of these three problem areas and they merit our 
concerted attention, but I point them up here primarily 
to clarify the proposition that even when good occupa- 
tional information is available, we must infer training 
needs relative to a specific set of behaviors, to be ap- 
plied in school or on the job, at a fairly precise point 
in the development of the individual incumbent. These 
three determinations must be made and clearly set forth 
as a matter of policy guidance. 

It might be mentioned that on-the-job training is 
almost always very difficult to administer, control, and 
evaluate and offers a promising application of pro- 
grammed teaching techniques as a supplement to in- 
struction received from supervisors. 

Once the educator knows the training objectives in 
terms of behaviors to be produced in individuals and 
where the training is to occur, as well as its temporal 
application, he is ready to infer from the desired be- 
havior statements exactly what it is which he must 
impart to the student. Notice that the ultimate test of 



I 



Trends In Programmed Insfrucflon 

his success will remain the same as his original objec- 
tive, which is the behaviors in the task list which are 
performed by the student after his training has been 
completed. In other words, there is invariance in the 
criterion, the behavioral task statement is the basis of 
determining training needs as well as the criterion for 
evaluating the training after it has been given. Inade- 
quacies in the training program may stem from either 



inaccurate inference of the training needed or from in- 
effective administration of the training. We might pro- 
vide the wrong training very eflSciently or the right 
training very poorly. With the task statement as a be- 
ginning point as well as an end-point, the difficulty 
of inferring training needs is reduced and it becomes 
possible to pinpoint the source of inadequacies in the 
training situation itself. 
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CONSIDERATIONS WHEN ORGANIZING PROGRAMMING PROJECTS 

Wiiiiam J. Knight 

William Knight discussed the problems of achieving standardization and quality of 
programmed materials and avoiding redundancy in the preparation of programs 
through proper organization. He concludes that centralization of programming is 
necessary until more qualified programmers are available to achieve proper decen- 
tralization of functions to achieve better customer service. 



INTRODUCTION 

As a member of management, I am concerned about 
the growth, of the Programmed Instruction (PI) move- 
ment. All of us connected with the movement should 
have one overall objective; make sure that the P.I. 
movement achieves the status of a major industry. One 
factor that will ensure this objective is the development 
of quality programs. 

In the last few years, most organizations developing 
programs realized that the programming task (writing 
frames) was only 25% of the total development cycle 
and even if done well did not necessarily mean a quality 
program would result. In many cases, this recognition 
has resulted in the development of better programs by 
exposing the effects different types of organizational 
designs can have on the quality of the program. 

This paper will explore the Centralized and Decen- 
tralized organizational designs for the primary purpose 
of examining the effect on program quality. 



ORGANIZATIONAL DESIGN AND 
PRODUCT QUALITY 

Figure I illustrates the structure of a centralized 
organizational design, i.e., one Training Division to 
satisfy all the PI requirement of the Military Electronics 
Division and the Commercial Electronics Division of 
KAL Industries. 

Figure 2 illustrates how the structure of KAL Indus- 
tries would change if a decentralized design were 
selected, i.e., both divisions have their individual au- 
tonomous training departments. 

Availability of qualified programmers/programmer 
trainees has been a major factor in the selection of 
an organizational design. It is quite obvious, there is 
no easy method to develop individuals who can even- 
tually do a creditable programming job in a reasonable 
amount of time. Because of this programmer shortage, 
many companies have been forced into the centralized 
design even though a decentralized design would be 
more efficient if the manpower were available. North 
American Aviation, Inc., a decentralized corporation. 
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initiated a programmer training project which included 
the training of several already qualified training spe- 
cialists. One reason why this project worked was be- 
cause the programmer trainees had training know-how 
before their PI training class started. 

After training many programmers over the past two 
years, Training Systems, Inc. (TSI) has found a high 
correlation of success between qualified trainers and 
successful programmers. This should not be surprising 
since a job analysis of the programming function speci- 
fies a requirement to know basic training principles 
in addition to programming principles. Too often in 
the past two years people have specified criteria for 
selecting programmers without first doing a detailed 
job analysis of what the programmer actually does 
when working. This has caused many people to believe 
that a writer (technical, fictional, etc.) could become 
an “instant programmer” as long as he liked to work 
in a closet for eight hours a day. It should be obvious 
to all training directors that a programmer must know 
the fundamentals of training or teaching. Once a writer 
has acquired a training knowledge the writer may be 
able to learn programming. This is also true for astro- 
nauts, engineers, accountants, secretaries, etc. 

The major advantages of a centralized organizational 
design (one Training Division, i.e., PI function) are 
as follows: 

1. Ease of control 

2. Better utilization of available programming man- 
power 

3. Uniformity of policies 



William J. Knight 

Ease of control occurs when the number of check 
points is decreased. 

In the decentralized design of Figure 1, Training 
Department (TDl) operates completely independent 
of TD2. Since the subject matter for commercial elec- 
tronics is similar to that of military electronics, the 
possibility of producing redundant programs is high. 
The centralized design of Figure 1 makes this redun- 
dant programming highly unlikely since there is only 
one Training Department. 

The uniformity of policy characteristics stems from 
the fact that all policies are made by one person. This 
means that the quality control standards will be con- 
sistent with the overall image of the company. It is 
quite possible that the two training departments shown 
in Figure A could have two different sets of testing 
procedures. One procedure might state that validation 
must be accomplished by using large numbers of stu- 
dents. The other procedure may use small numbers 
for validation. This inconsistency cannot occur in a 
centralized organization. 

CONCLUSIONS 

At present, and mainly because of the labor shortage, 
a centralized organization can best guarantee the pro- 
duction of quality programs. As more trained person- 
nel become available, decentralized designs may be a 
more efficient and effective method of developing pro- 
grams. Time will teU. 
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Vli Military and Governmental Applications* 

PROGRAMMED INSTRUCTION BREAKTHROUGH IN AIR FORCE TRAINING?? 

James E. Briggs, Lt. General, USAF (Retired) 

The USAF Air Training Command, which General Briggs commanded at the time this 
talk was presented, is probably the largest training institution in the world. As such 
it is constantly on the alert for new methods of training more men better, cheaper, 
and faster. In an effort to improve training technology they have trained more than 
300 programmers and have already achieved savings in training by this experimental 
effort. This has been described as “the largest of all automated teaching programs.” 

At the same time the Air Force has conducted familiarization courses for managers so 
that any traditional resistance to change might be alleviated. Early results in the areas 
for which programs were produced were most gratifying both in cutting training time 
and increasing amount of material learned. Although the final evaluation of pro- 
grammed instruction as the breakthrough in training technology has not yet been 
made, the Air Force is committed to give it a fair trial. This is in iine with the prec- 
edent set by all the services in taking the lead in experimenting with new training 
technologies. 



The mission of the Air Training Command is ex- 
pressed by its motto, “Prepare the Man.” Our function 
is to meet the Air Force’s training requirements for a 
myriad of weapons systems. Not many years ago we 
could turn out an electronics technician for the F-80 
with only ten weeks of training. A squadron needed 
seven of these men. For the F-106 squadron, we need 
70 electronics technicians, and most of them must 
spend more than a year in electronics training before 
they carry their own weight. Furthermore, F-106 sup- 
port men in five specialities must be taught basic elec- 
tronics for their jobs. 

Examples like this could be quoted for all the tech- 
nical fields used in the Air Force including the missiles, 
the communications equipment, and the many subsys- 
tems that make them work. Each advance increases 
training time by requiring that more things be taught 
and learned which in turn increases the cost of training 
and trmning equipment. 

Our job involves people — and it’s a big job. Last 
year we trained over 362,000 airmen at a cost of 
approximately 700-million dollars. As new coimnand 
and control systems approach operational status, our 
training requirements will become more complex. We 
have made many improvements in our training pro- 
cedures and management techniques, but we are con- 
stantly concerned with the Command’s ability to meet 
the Air Force training job of the future. We must 
constantly revise our training concepts, methods, and 
procedures to meet tomorrow’s mission in terms of cost. 



* Editor’s note: The USAF programs mentioned in this section 
are experimental and developmental in nature and are not at 
this time available for general distribution. 



timely production of trained personnel, and quality of 
graduates. 

Our concern with these future problems caused us 
to establish what we called our Path Finder Study 
Group in October 1961. This was a group of senior 
Air Force oJQficers, who conferred with some of the 
most outstanding civilian scientists in the country dur- 
ing their study. This group thoroughly examined some 
of the technological advances affecting our training 
mission. In doing so, they identified the need for a 
number of changes in our methods of operation. These 
changes included a shift in our training capability from 
primary dependence upon resident technical schools to 
a more evenly balanced system of resident and exten- 
sion service. Another change was the development of 
a management system that would provide faster reac- 
tion in controlling the student flow. Still another change 
required us to develop training technologies centered 
on individual learning rates rather than on group learn- 
ing rates. 

Prior to the Path Finder Study Group’s recommen- 
daitons, we had begun to investigate the possibilities 
inherent in programmed instruction in the Air Training 
Command environment. Path Finder not only endorsed 
our action but encouraged us to expand our experi- 
mental effort. We decided to accomplish this investi- 
gation in two phases. Phase I was to be an experi- 
mental application of programmed instruction to a 
select and heterogeneous group of courses, and Phase II 
would consist of expanded use, if the results of Phase I 
warranted such action. Our Phase I activities will, soon 
be completed. Our preliminary results, however, have 
been most gratifying. Under our concept of concurrency, 
we already are establishing plans to implement Phase II. 



Trends In Programmed Instruction 

We feel that this action will be more than warranted 
by the results that we have already received from 
Phase I. 

Our most immediate problem in establishing pro- 
grammed instruction was to develop an in-house capa- 
bility. Since November 1961, the Air Training Com- 
mand, through contract with outside agencies, has 
trained more than 300 personnel throughout the Air 
Force in the techniques of developing programmed 
materials. Although it has been said that this effort 
more than doubled the number of instructional pro- 
grammers in this country, our capability is still minimal 
when one considers that we have more than 2400 train- 
ing courses. My advisors inform me that, were we to 
exploit our present capability to its maximum, we could 
not program more than 5% of the Air Training Com- 
mand effort within the next three years. 

In addition, realizing that we would encounter the 
usual resistance experienced by those introducing a 
relatively new instructional methodology, we provided 
orientation and indoctrination in programmed instruc- 
tion to more than 1500 supervisors and training man- 
agers. This effort was very important in producing the 
cooperation and support necessary to an evaluation 
of this effort. 

At present, we have more than 100 programmed 
instructional packages in use or in the process of being 
developed. These materials cover subjects ranging from 
“T-37 Emergency Procedures” in flying training to 
“Precision Measuring Equipment” in technical training. 

Our general objective is to determine whether a pro- 
grammed instructional package in lieu of, or as a sup- 
plement to, conventional methods of training will sub- 
stantially improve the quality of our graduates and 
materially decrease the costs. Concretely, we have 
specified that for a program to be acceptable, 90% 
of the students must score 90% or above on a compre- 
hensive test covering all of the course objectives. 

Our preliminary results show that we are on our 
way to reaching our 90/90 objective, with reductions 
in training time of 25-50% . One of our programs has 
reduced our training time as much as 83%. At Cha- 
nute Air Force Base, for example, we programmed a 
six-hour block of instruction in “Basic Hydraulic and 
Pneumatic Principles.” This programmed course took 
only one hour on the average for students to complete 
and, at the same time resulted in a gain in performance 
of 20%. At Keesler Air Force Base, programming 
reduced the training time for an introductory AC&W 
radar course from fifteen hours to five hours, with 95% 
of the students scoring over 85% on the criterion test. 

A program developed at Amarillo Air Force Base on 
the use of hand tools raised the average level of 
achievement by 33% with a 40% reduction in training 



time. The test for this package was a job performance 
test — ^the actual use of hand tools in a realistic job 
environment, not a paper-and-pencil exercise. At Shep- 
pard Air Force Base, a program on reading and inter- 
preting electrical diagrams is producing students who 
score 16% higher than those taking conventional in- 
struction and with an average saving of 36% in train- 
ing time. 

Such decisive results have caused us to look very 
hopefully to our Phase II efforts. 

We have not been content to simply develop our 
in-house capability. We have tried to exploit as many 
facets of programmed instruction as the current state 
of the art would permit. Our training centers have re- 
viewed off-the-shelf programmed materials relevant to 
their training mission. We cannot afford the luxury 
of duplication. 

Another area we have been exploring for the use of 
programmed materials is in our on-the-job training 
programs. The frequent equipment modifications in 
our complex defensive and offensive weapons systems 
require constant training of our personnel. Since many 
of these technicians are in remote or inaccessible areas, 
programmed materials could be produced centrally 
and made available before equipment modifications 
actually occurred. At least one programmed on-the-job 
training package is under development at each training 
center. 

I could go on to describe our many other activities 
related to the exploitation of programmed instruction, 
such as our effort to bring together programmed in- 
struction and instructional TV, the use of synchronized 
audio-visual devices, and many others. 

Although this methodology is stiU young and many 
problems associated with its use are yet unsolved, we 
are certain that it will have many applications to our 
training requirements. We. do not, of course, claim that 
programmed instruction is. the answer to all our prob- 
lems. We do feel that extensive use of this technique, 
under appropriate conditions, will help us lo meet our 
training requirements faster, with less cost, and at the 
same time provide the using commands with more 
thoroughly trained personnel. We feel that programmed 
instruction many result in the emergence of a true edu- 
cational technology with valuable implications for edu- 
cational institutions, industrial organizations, and the 
armed services. 

Since World War II, the armed services have spear- 
headed movements in instructor training, audio-visual 
devices, closed and open-circuit TV, and language lab- 
oratories, to mention only a few of the techniques 
which have been explored. We shall continue to in- 
vestigate any technology that may alleviate the training 
problems created by technological growth. 
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We have under way plans to establish a model ex- 
perimental classroom for conducting pilot studies and 
demonstrations of new educational techniques. We can 
use this classroom to evaluate automated teaching and 
audio-visual devices, to demonstrate classroom manage- 
ment problems, and for instructor training. It will also 
provide facilities for controlled experimentation in vari- 
ous instructional techniques. 

Even beyond these continuing developments, we 
must remain alert for future breakthroughs in educa- 
tional technology. We rely on, and will continue to 
maintain close liaison with, educators, research institu- 
tions, industrial concerns, and the other armed services. 
We realize that all of these groups are actively con- 
cerned with these problems. 

It is obvious that projected command and control 
systems will create problems in the scope and com- 
plexity of training. As automation and systems com- 
plexity increases, the tasks assigned to human control 
increase in difficulty. Our task is to train the man to do 
his job efficiently. 

Will programmed instruction be the breakthrough in 
Air Force training? I don’t really know as yet. I do 
know this, however; that nothing has appeared on the 
horizon which has given us a more hopeful sign for 
such a breakthrough as has programmed instruction. 

It was Ralph Waldo Emerson who said that the re- 
ward of a thing well done is to have done it. I am not 
saying that we have already done it. The next year or 



James E. BriggS/ Lt. General/ USAF (Retired) 

two will demonstrate, more than anything I can say 
here today, whether we have done it or not. We are 
convinced that we have conclusively demonstrated that 
it can be done. WJiether it will be done or not is going 
to depend to a great extent on the willingness of many 
of our training managers and training staff to find ways 
of implementing those things that we have demonstrated 
could be done. This is, in and of itself, no easy task. 
I do not feel that we can confidently erase the question 
marks which follow the words “programmed instruc- 
tion” and “breakthrough” until we can look around us 
throughout the Air Training Command and find it a 
common sight to see students going through a variety 
of programmed materials, each proceeding at his own 
optimum pace, each achieving his own maximum 
quality performance in the shortest possible period of 
time with the minimum assistance needed from a master 
instructor, and all at a minimum of cost. Even when 
this is a common sight, we must still continue to be 
alert to new ways, new methods, and new ideas. 

Whether programmed instruction does or does not 
provide us with the promised breakthrough remains to 
be seen, but out of our efforts in experimenting with 
it has come a new appraisal of our training efforts, a 
new way of looking at the wide question of how should 
we best go about doing our training task. This, I am 
confident, will remain with us and for this I am 
grateful. 
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MECHANICAL TRAINING WITH THE K-38 REVOLVER 

Jackie Lang, TSgt, USAF and J. E. Melton, TSgt, USAF 

In this paper. Sergeants Lang and Melton discuss their efforts in developing an in- 
structional program for teaching both verbal and motor skills. The program they pre- 
pared resulted in a 35% reduction in training time, with trainees achieving better 
than 98% of the seven comprehensives verbal and motor skill obfectives. 



When the authors began progranuning the mechan- 
ical training package for the K-38 Revolver, we imme- 
diately looked for guidance in existing linear pro- 
grammed packages. A multitude of programs were 
available whose terminal behaviors were the listing of 
verbal terms and concepts, but there was no guidance 
in the form of published programs whose terminal be- 
havior involved non-verbal, motor or performance of 
skills. 

How does one program the performance of a skill? 
As neophytes to programmed instruction we welcomed 
any and all suggestions, for ours was a difficult task. 
We should like to share with you some of the problems 
we encountered in performing this task of programming 
verbal and non-verbal terminal behaviors. 

Before discussing the problems which arose, let us 
look at the objectives of the program for mechanical 
tr ainin g for the K-38 Revolver. The objectives range 
from the simple naming of parts to the performance of 
a manual skill. Specifically, a summary of the objec- 
tives is as follows. Upon completing the program, the 
student should be able to: 

1. Identify and name each of the 30 parts of the 
K-38 revolver 

2. Describe the mechanical function of each part 

3. Disassemble and assemble the revolver 

4. Clean and lubricate the revolver 

5. List the corrective action to be taken if a mal- 
function occurs 

To prepare the student to display these behaviors 
we were allotted four hours. This was the same amount 
of time allowed for traditional instruction. Here are the 
techniques we used and the problems we encountered 
in programming verbal behaviors. 

The revolver consists of thirty separate parts. We 
had to prepare the student to recognize each of the 
parts by its shape, to name each part, and to describe 
its function. The student was introduced to each part 
by referring him to an 8xlOV^" illustration, which de- 
picted and named that particular part along with three 
other parts. We then stated the function of the part 
and required the student to respond overtly to incom- 
plete rules and examples. 

Ori^nally the program was filmed for use in the 
Dukane Redi-Tutor teaching machine. The illustrations 



were placed in a separate booklet which required the 
student to shift constantly from machine to booklet. 
This wasted too much time. It also permitted easy ac- 
cess to heavily cued introductory illustrations during 
criterion frames. We then decided to publish the pro- 
gram in textbook form for further testing. The illus- 
trations were included in the textbook and were far 
more satisfactory. 

The Ruleg System was used throughout the program. 
The student was often required to view diagram, and 
other illustrations to complete the responses. These 
illustrations constantly exposed him to the parts loca- 
tions and their shapes. Criterion frames require the 
student to name a part, describe its function, and iden- 
tify the part in a panel. Diagrams and illustrations were 
used extensively in this program. 

In hopes of conserving material, a separate response 
booklet was prepared. In addition to increasing the 
length of the text, use of a separate booklet decreased 
the effectiveness of the program since a vast amount of 
time was wasted leafing through both of the books. The 
program was later revised so that the responses could 
be written in the text. 

Although considerable success was realized through 
this type of progra mming , the students encountered 
some difficulty when asked to describe the function of 
the revolver’s internal parts. To strengthen this part of 
the program, we realized that the actual weapon must 
be included. Use of the K-38 revolver permitted the 
student to observe the mechanical function of each 
actual part in relation to other parts. For example, 
after the student was conditioned to respond, “the hand 
rotates the cylinder,” he is then told, “cock the weapon 
and observe the hand’s movement in performing its 
operations.” Use of the actual weapon may not follow 
any known linear programming techniques, but we 
found it effective. Students were able to respond cor- 
rectly during criterion testing. 

Use of the weapon was also necessary in the pro- 
gramming of non-verbal behavior such as the actual 
disassembly and assembly of the weapon. However, the 
revolver was not used in the early developmental stages 
of this portion of the program. Some of the students 
were able to demonstrate disassembly and assembly 
skills although the weapon was not introduced until 



testing. To insure that all the students could perform 
the task, the revolver was included early in the pro- 
gram. 

Use of the weapon presented problems. Students 
were frequently required to write a response and per- 
form a task in the same frame. For example, a frame 

would read, “turn the front side place screw 

The responses desired were, first, write the response, 
“counter clockwise”, second, physically turn the screw. 
This type of frame disturbed the student since he had 
to decide which action to perform first. Though the 
problem seems minor, it caused us much grief. It was 
finally resolved by writing a separate frame for each 
of the responses. Another problem was that frames 
requiring students to perform a task lacked reinforce- 
ment. For instance, when the student is conditioned to 
respond “the first part to be removed in revolver dis- 
assembly is ‘the front side plate screw’”. The next 
frame merely tells him to perform this step. 

It is after this performance step lack of reinforce- 
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ment occurs. To strengthen responses of this nature, 
illustrations were included to show the part that should 
have been removed and what the revolver should look 
like after the part had been removed. This simple step 
seemed to provide the necessary reinforcement. Once 
the actual revolver was utilized in the program and 
manual skill responses were reinforced, the program 
was highly successful. 

If there has been one significant finding in program- 
ming of this type, it is this; The tools and equipment 
which are required to perform a task must be included 
in a program to insure its success. 

Upon the completion of the development of the 268 
frame linear text as described above, the program was 
field tested under operational conditions on three sepa- 
rate groups of trainees totalling 53 students The pro- 
gram resulted in a 35% reduction in training time, 
from 150 minutes to 98 minutes, with trainees achiev- 
ing better than 98% effectiveness on both the wiitten 
and performance criteria tests. 
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AN EXPERIMENT IN PROGRAMMING THE CARE AND USE OF 
AIRCRAFT MECHANICS’ HAND TOOLS 

Lewis L. Coleman, SSgt, USAF 

SSgt L. L Coleman describes what has been considered the most popular pro- 
grammed text in the Air Training Command study of experimental programmed ma- 
terials. This program not only teaches nomenclature and the application of hand tools, 
but also includes practical experience in the use of bond tools in the simple tasks on 
the jet engine. The experimental group showed gains over the conventionally taught 
control group on both the written and performance tests. 



In the Jet Engine Mechanic Course at Amarillo Air 
Force Base, instruction in the care and use of common 
hand tools is introduced in the second day of the 
course. Instruction includes nomenclature, physical 
characteristics, care, and application of hand tools 
such as wrenches, pliers, screwdrivers, sockets, thick- 
ness gages, torque handles, etc. Instruction is of the 
conventional type utilizing discussion, demonstration, 
two training films, and a student text. Training also 
includes practical experience in using about 50% of 
the tools by performance of simple mechanical tasks 
on a jet engine. 

This instruction was considered satisfactory, but an 
experiment was conducted to see if the training objec- 
tives could be more effectively accomplished by use of 
an automated instructional program. 

A 302 frame linear-type program was developed 
with the following stated objective: The student will 
be able to care for and use the hand tools commonly 
used by a jet engine mechanic. 

About 75 % of the frames contained illustrations. 



The nut below wiU tighten since it is being turned 



(Illustration) 



All the nuts and bolts on jet engines are right 
hand thread. They turn clockwise to tighten and 
to loosen. 



Use the hinge handle and socket to loosen nut 
again. Retorque to 185 inch pounds. Your set- 
ting should look like this: 

(Illustration) 



Using a hinge handle, extension, and V 2 inch 
socket, loosen the nut in area “H”. 

(Illustration) 



To provide practical experience for the students in 
selecting and using tools, eight identical trainers were 
designed to be used in conjunction with the program. 
Throughout the package there are frames directing 
the students to select and use tools to accomplish sim- 
ple tasks provided by these trainers. 

Evaluation of the program was accomplished by 
comparing results of written and performance tests 
administered to a control group of 52 students receiv- 
ing conventional instruction and an experimental group 
of 52 trained by the program. Students participating in 
the evaluation project were regular input airmen having 
an age range of 17 to 22 years and an average edu- 
cation of 11 years. The two groups were carefully 
matched by selection on the basis of mechanical and 
general aptitude scores, and by educational level. The 
matching of the two groups was within 99.8% accuracy. 

The Jet Engine Mechanic Course has a weekly 
student entry which necessitated the use of seven classes 
to provide a total of 104 students. As each class 
entered training, it was divided to form the control and 
experimental grouping with approximately eight stu- 
dents in each group. During the period of seven weeks 
the control groups were taught in the conventional 
manner. Each class had a different instructor. 

The programmed instruction experimental groups 
of approximately eight men each were trained in a 
separate area. Each week a monitor was assigned to 
the experimental group to provide each student with a 
programmed text, a trainer and a tool kit. The moni- 
tor also maintained a record of the time required for 
each student to complete the program. The students 
were given no assistance in using the program. 

There were two tests used in the evaluation, a fifty 
question written test containing verbal conceptual items 
and a performance test covering skills in using tools. 

The two tests were used as a pre-test one day before 
training, as a post-test one day after training and again 
for retention after 30 days. The tests were adminis- 
tered and scored by two instructors and two clerks who 
were associated with neither the conventional instruc- 
tion nor the programmed package. During testing, aU 



students in both groups were intermixed so that the 
examiners did not know to which group the students 
had been assigned. 

The statistical results of the testing are as follows: 



COMPARISON OF CONTROL GROUP 
& EXPERIMENTAL 


Pre- 


Post- 


Test 


Test 


Control Group 52 Students 6-63 


45-77 


Experimental Group 52 Students 6-56 


59-98 


Group written pre- and post-test results, 
score 100. 

Table 1 


Possible 



COMPARISON OF AVERAGE TIME 

Control Group 52 Students 5 Hrs 10 Min 

Experimental Group 52 Students . . 3 Hrs 21 Min 



On the written pre-test the control group averaged 
32.5% as compared to 32.7% for the experimental 
group. On the post-test, however, the control group 
averaged 63%, a gain of 93.9% as compared to the 
experimental group which attained 84.2%, a gain of 
157.7%. 
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The performance test showed smaller differences be- 
tween the groups. The control group averaged 61% 
on the pre-test and 82% on the post-test, a gain of 
32.2%. The experimental group averaged 59.2% on 
the pre-test and 86.8% on the post-test, a gain of 
46.4%. 

Both groups were tested with the same tests 30 days 
after training to determine retention. The results of 
the retention testing are not considered valid because 
hand tools were used repeatedly during the 30-day 
interval. A slight gain in retention scores of the con- 
trol group indicates additional learning was possible. 
The very slight drop by the experimental group was 
from a higher grade plateau with a higher total net 
gain beyond the pre-test scores. 

The time required to teach “Hand Tools” by the con- 
ventional method is 5 hours and 10 minutes. The 
average time to complete the programmed package was 
3 hours and 5 minutes with a time spread of 2 hours 
to 4 hours between the fastest and slowest students. 

After completion of the evaluation of this program, 
the package was revised and further testing shows that 
almost nine out of ten students reach over 91% of the 
training objectives. 

From this experiment we can conclude that this 
programmed package can teach knowledge items more 
effectively than conventional instruction and also ac- 
complish the instruction in less time. Of particular 
significance is the fact that a programmed package can 
very effectively train students in simple manual skills. 
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THE AIR FORCE LEHER 



John D. Hathaway, MSgt, USAF and Dannie M. GHIham 

John D. Hathaway and Dannie M. Gillham were trained as programmers during 
Phase I of the Air Force’s extensive program development project. They wrote a pro- 
gram designed to teach basic Airmen both the knowledge and skills'required to type 
a mailable Air Force letter. This paper describes the procedures they used in evaluating 
and revising the program through three classroom tryouts. They have much to offer 
in the way of practical suggestions. 



The Air Force Letter program is designed to replace 
an eleven hour lesson in the Administrative Specialist 
Course. Generally speaking, the program teaches the 
same material now taught conventionally in the course. 

The first question to be answered was whether the 
performance part of the lesson would be programmed. 
In the conventional course, three or four class periods 
are devoted to presenting the academic portion of the 
lesson and the remaining seven or eight periods are 
used by the students to type the actual typing problems 
listed on their work sheets. The authors could have 
programmed only the information needed by the stu- 
dents to type the work sheets. In this case, the students 
would have begun the work sheets as soon as they had 
completed the program. 

A decision w«.t made to attempt to teach the whole 
lesson in one program. Once the decision was made to 
program the entire lesson, none of the authors would 
settle for any type of criterion test that did not require 
the student to type a mailable Air Force letter. The 
test devised consisted of three problems which required 
the student to demonstrate his ability to type an Air 
Force letter on both preprinted and plain stationery 
and a second page continuation for either type of 
stationery. 

At this stage of the project the lesson was outlined. 
This outline was made in detail so that very little 
additional work was necessary when the writing of the 
frames was begun. 

For the first try at writing frames, each of the three 
authors attempted to write one third of a particular 
topic. This system was rejected because when the three 
individually developed sets of frames were brought 
together it was very difficult to fit them into a mean- 
ingful whole. The system finally used called for each 
writer to devise a set of frames for the same small seg- 
ment of the program. When all were ready, a joint 
session was used to discuss the three sets of frames and 
decide on the best mode of attack. Then, the best parts 
of all three were combined into the final program. 

This method prevented any one writer from getting 
too far afield before being drawn back on the track 
by the other two, while still providing several sets of 
ideas on each segment. 



As soon as each main segment of the program was 
finished it was tried out on individual students. Luckily, 
the student obtained for the first trial run was quite 
bright and very articulate. As he went throng the 
frames which had been set up on 3 x 5 cards he made 
many helpful comments. He suggested use of simpler 
terms, pointed out gaps in the material presented, and 
indicated military jargon not familiar to most of the 
students. After this trial, a number of changes were 
made in the program. 

An average student was used for the next tryout. 
At this time, the first two main segments of the pro- 
gram were tested. This student was not as helpful as 
the brighter, first student but still made some worth- 
while suggestions which were used to improve the 
program. 

A student from the lowest portion of the class was 
used for the third tryout which consisted of the first 
three main segments. Communication with this type 
of student was much more demanding. It was difficult 
to gain his confidence and to draw out his comments. 
The main accomplishment was to identify frustration 
frames. Although an attempt was made to determine 
what caused the frustration, in most cases the student 
was unable to identify and explain the trouble. How- 
ever, these frames were worked over by the authors 
with an eye toward simplification and better continuity. 

This general process was continued until the entire 
program was completed. Once the entire program 
had been tested, several students were used in indi- 
vidual tryouts. At this time arrangements were made 
for the group tryouts. 

During the period when the final arrangements were 
being made for the trials, a good deal of time was 
spent in working out the ground rules under which 
the trials would be conducted. Some of the stipulations 
were as follows: 

1. No student in the control or experimental groups 
would be permitted to attend remedial training during 
this period. 

2. No student of either group would be permitted 
to take any training materials out of the classroom at 
the end of the shifts. 



3. The instructors in the experimental classes could 
give no help to the students — ^the program was to stand 
alone. 

An instruction booklet was designed for the experi- 
mental class instructors, giving detailed instructions as 
to how the class was to be conducted. A similar, 
though somewhat smaller booklet, was designed for 
the control class instructors. These were similar to the 
type of instructions used to administer Air Force tests 
such as the Airman Qualifying Examination. 

The student population consisted of both male and 
female airmen who had been in the Air Force for only 
a few weeks. Their average age was 19 and most of 
them had graduated from high school. They attended 
school on one of tliree six-hour shifts which began 
at 6 A.M., 12 Noon, and 6 P.M. 

The students in the experimental group averaged 
82% correct frame responses. On the post-test, the 
experimental group averaged 59 correct answers out 
of a total of 78 raw score points or 76% while the 
control group average was 67 out of 78 or 86%. 

These results caused the authors to take another 
long look at the program. Although all frames were 
reviewed, emphasis was placed on those to which less 
than 90% of the students responded correctly. Some 
material was deleted from the program entirely in 
cases where it wasn’t taught in the conventional coarse 
and was not useful in ordinary daily work. Some 
frames were arranged in a different sequence; some 
frames were added to close gaps in the presentation 
of the material; some terms were simplified; some awk- 
ward sentences were rearranged, etc. 

Knowing that the first tryout would turn up some 
needed changes, the lack of control in the testing situa- 
tion was ignored. However, certain changes were 
made in the tryout procedure prior to the second try- 
out. The criterion test was revised because it was dis- 
covered that the manner in which the problems had 
been stated was causing misinterpretation by both the 
conventional and program students. 

For the first trial, the classes which would take the 
post-test after having been taught by the conventional 
method were identified early. The instructors were 
briefed as to the ground rules and told that the experi- 
ment was being conducted to determine the effective- 
ness of the program in contrast to the conventional 
methods they had been using. Course personnel were 
aware of the content of the post-test for some time and 
the post-test was in the course building for six or 
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eight hours before the test was to be given. Grading 
was done by course personnel. 

The second group tryout was performed in such 
a manner that the conventional instructors did not 
know that their classes were to take the post-test until 
the day the test was given when the tests were delivered 
to the course. This time the students ir the experiment- 
al group averaged 93% correct frame responses. On 
the post-test, the experimental group had an average of 
39 out of 47 total raw score points or 83%, while the 
control group average was 42 out of 47 or 89% . This 
was an increase of 7% for the experimental group but 
only 3% for the conventional group. It appeared that 
the program was now teaching about as effectively as 
the conventional method. 

A review of the response sheets for the frames in- 
volving typing indicated that the students were unable 
to evaluate their work effectively. Due to this finding, 
another tryout was arranged with the provision that 
the instructor help the students as needed with special 
emphasis on the first of the nine complete letters that 
the students type in the two booklets of frames con- 
taining the performance part of the program. The test 
results of this class were better. The students in the 
experimental group averaged 95% correct frame re- 
sponses. On the other post-test, the experimental group 
had an average of 43 correct answers out of total of 
47 raw score points or 91% while the control group 
average was 42 out of 47 or 89%. At this point the 
programmed instruction package was perfonmng very 
slightly better than the conventional method. 

In the conventional classes, eleven 50-minute periods 
are devoted to the Air Force letter. This is 9 hours and 
10 minutes total time. The experimental group on the 
first group tryout took an average of 9 hours to finish 
the program. In the second group tryout, the average 
time was 8 hours and 13 minutes. In the third group 
tryout the average time was cut to 7 hours and 28 
minutes when the instructor was permitted to assist the 
students. 

The following three conclusions seem to be appro- 
priate, based on the results of this experiment: There is 
an advantage to permitting the instructor to help the 
students. Rigid controls are necessary in order to ob- 
tain meaningful results from iments. Existing 
manuals can be used to advantage in devising programs 
by reducing the number of frames required and con- 
comitantly helping the students become familiar with 
the use of the manual on the job after they leave school. 
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PROGRAMMED INSTRUCTION AT CHANUTE AFB, ILLINOIS 

Robert F. Harris 

Robert F. Harris outlines the mission of Chanute Air Force Base and discusses the 
application of programmed instruction in comparative tests with conventional instruc- 
tion. The author lists the comparisons which show significant increase in student 
achievement at the same time training time is substantially reduced. 



Chanute Air Force Base is located in East Central 
Illinois approximately 120 miles south of Chicago and 
approximately 20 miles north of Champaign, Urbana, 
the home of the Universtiy of Illinois. Chanute Air 
Force Base, Illinois, is known throughout the United 
States Air Force and the armed forces of many coun- 
tries. The 3345th Technical School at Chanute con- 
ducts over two hundred resident courses, and it has 
a field training team of approximately 600 instructors 
who take training to specialists at their permanent 
duty stations. The Technical School has a staff of 
technical writers who prepare an equally significant 
volume of manuals and instructional materials for use 
by USAF officers and airmen as a part of their on-the- 
job training and self-improvement programs. 

The scope of the training program at Chanute is 
diversified to an extent that graduates may be assigned 
to any one of the major air commands of the Air Force. 
In operation at the present time is a diversified and 
complex group of courses that provide individual train- 
ing for officers, airmen and Air Force employees who 
will become members of the personnel sub-system for 
such weapon systems as the B-52, KC-135, B-58, 
GAM-77 IM-99 and the Minuteman. 

With the preceding brief description of the environ- 
ment in which we have worked with programmed in- 
struction, the remainder of my comments are about 
how we have used programmed instruction and on 
certain observations we have made during our limited 
experience with it. 

The first professional formal training for our per- 
sonnel selected to become programmers started in 
January 1962, just a little over a year ago. Training 
for some of our programmers started as recently as 
September 1962. 

In the absence of establi.shed and universally ac- 
cepted criteria for selection of programmer-trainees, 
approximately twenty-five individuals were selected, 
on a purely subjective basis, to receive the training. 
The recommendations of the individual’s supervisor 
and such guide lines as retainability, professed interest 
in the new techniques, and knowledge in certain techni- 
cal areas in which units of instruction were to be pro- 
grammed were observed to the extent practicable. The 
list included offcers, airmen and civilian personnel. 



The selection of units of instruction for the pro- 
grammers to program was a cooperative effort that 
involved personnel from Headquarters Air Training 
Command, the Chanute Technical Training Center, 
the School, The School Department and the pro- 
grammer-trainee. A list of the units of instruction to 
be programmed is shown on Chart 1. This is a list 
of Programmed Instruction Packages (PIP’s) that we 
at Chanute refer to as our prototype or first genera- 
tion PIP’S. 

Our prototype PIP’s were or are being developed un- 
der a team concept in which two or three programmers 
serve on a team responsible for programming a given 
unit of instruction. For comparison purposes, the sec- 
ond generation PIP’s are to be written with just one 
programmer assigned on each team and he will be 



Units of Instruction to be Programmed 
PIP: 

Basic Hydraulic & Pneumatic Principles, 6 hrs. 
Linear 

Property Responsibility, 8 hrs. Linear 
Principles of Pitot-Static Systems, 9 hrs. Linear 
IP 42350, Part A: Warning Systems; Part B: 
Batteries & Their Maintenance 

Gyroscopic Principles, 4 hrs. Linear 
Aircraft Familiarization, 4 hrs. Linear 
How to Study & Principles of Learning, 16 hrs. 
Linear 

Aerodynamics, 6 hrs. Branching 
Aircraft Oxygen Systems, 6 hrs. Branching 
Observing, Recording & Encoding Visibility & 
Obstruction to Vision, 7 hrs. Branching 
Interpolation for Determination of Dew Point, 
7 hrs. Branching 

Maintenance Data Collection Reporting, 17 hrs. 
Linear 

General Principles of Test Construction & Con- 
struction of Multiple-Choice Items, IV 2 hrs. 
Linear 

Principles of Automatic Transmissions, 12 hrs. 
Mathematics 

Jet Engine Compressor Section, 6 hrs. Linear 

Chart 1 



encouraged to consult and counsel with other pro- 
grammers as the need arises. In referring to our second 
generation PIP’s, we refer to a list of units of instruction 
to be programmed by a programmer who has served 
on a team of two or three programmers that has com- 
pleted a prototype PIP. 

Several of our first generation PIP’s have been com- 
pleted to the initial satisfaction of the programmer 
and are being used in a comparative study. In the 
comparative study we are comparing the performance 
of a control group of students with the performance of 
an experimental group of students. The control group 
has received instruction on the unit of instruction by 
the “conventional” method and the experimental group 
has received instruction on the same unit of instruction 
by PIP. The control group and the experimental group 
for a given unit of instruction are administered the same 
pre-test, post-test and retention test. In each case, the 
tests are composed of items mutually agreed upon by 
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the conventional instructor, the programmer and an 
instructor supervisor. Chart 2 contains data that are 
typical of data collected in the comparative studies 
of our first generation PIP’s 

In our second generation PIP’s, greater emphasis is 
being placed upon these three functions: 

1. The objectives for the unit of instruction, must 
be stated in measurable terms of observable student 
behavior, developed as a team effort and released to 
the conventional instructor and programmer alike by a 
training manager. The team that develops the objec- 
tives for the unit of instruction must include a training 
supervisor and a programmer. 

2. A criterion test must be developed and approved 
by a training manager. The criterion test must be com- 
prehensive, testing the student’s performance on each 
of the behavioral objectives. 

3. Given a list of behavioral objectives and a com- 
prehensive criterion test, both of which have been 



Chanute TTC 
Tryout Data 





PIP 


Conven- 

tional 




PIP 


Conven- 

tional 


Pitot Static Systems 






Pre-Test Scores 


41% 


38% 


Linear, 297 Francs 






Post-Test Scores 


83% 


73% 


Students 


43 


46 


Range of Post-Test Scores 


53-97% 


47-90% 


Pre-Test Scores 


26% 


28% 


Average Time 


2:46 


3:30 


Post-Test Scores 
Range of Post-Test Scores 
Average Time 


92% 

62-100% 

3:05 


71% 

38-92% 

6:45 


How To Study 
Linear, 328 Frames 
Students 


48 


41 


Aircraft Oxygen System 






Pre-Test Scores 


27% 


19% 


Branching, 217 Frames 






Post-Test Scores 


92% 


60% 


Students 


52 


48 


Range of Post-Test Scores 


80-100% 


37-83% 


Pre-Test Scores 


54% 


55% 


Average Time 


2:39 


4:30 


Post-Test Scores 
Range of Post-Test Scores 
Average Time 


88% 

78-100% 

4:30 


77% 

22-84% 

4:30 


Principles of Learning 
Linear, 289 Frames 
Students 


48 


41 


Basic Hydraulic and Pneumatic 




Pre-Test Scores 


20% 


22% 


Principles 






Post-Test Scores 


90% 


56% 


Linear, 126 Frames 






Range of Post-Test Scores 


66-100% 


20-91% 


Students 


38 


40 


Average Time 


3:35 


7:30 


Pre-Test Scores 
Post-Test Scores 
Range of Post-Test Scores 
Average Time 


57% 

82% 

53-100% 

1:16 


55% 

75% 

30-80% 

4:00 


Aerodynamics 
Branching, 201 Frames 
Students 
Pre-Test Scores 


26 

63% 


20 

67% 


Gyroscopic Principles 






Post-Test Scores 


97% 


94% 


Linear, 237 Frames 






Range of Post-Test Scores 


81-100% 


73-100% 


Students 


47 


47 


Average Time 


2:27 


4:30 



Chart 2 



Trends In Programmed Insfruciion 

approved by a training manager, the programmer 
prepares the PIP. The standard of performance for the 
PIP is that at least 90 % of the target population will 
achieve at least 90% of the objectives. 

In some respects, our use of programmed instruc- 
tion so far might be described by the casual observer 
as an effort that has produced results that are less than 
spectacular, and to those of us who have worked 
closely with the programming project, the experience 
that we have had with our first generation PIP’s has 
been very satisfying and stimulating. Although mis- 
takes have been made, not one of our first generation 
pip’s will be considered a failure by the Chanute 



Technical Tr ainin g Center programmer team. In each 
mistake that was made, there was a lesson to be learned 
by someone. In most cases the lesson that was learned 
can be applied in our job of improving the experiences 
of students whose behavior we seek to influence, shape 
or change. 

One final point: We are convinced that our study 
and limited use of programmed instruction at the 
Chanute Technical Training Center must be considered 
a success. The credit for the success of this project 
goes to the diligent, determined, devoted, skillful and 
patient programmers with whom it has been my privi- 
lege to work. 



PROGRAMMED INSTRUCTION AND THE INSTRUCTIONAL SYSTEM 
AT THE U.S. AIR FORCE ACADEMY 

Arthur R. Steiger, Major, USAF and Franklin C. Sutler, Jr., Major, USAF 

The Air Force Academy has concluded that a system of Instruction, developed from 
a thorough analysis of: (1} course objectives, (2) learning processes, (3) media and 
materials, (4) methods of presentation, (5) evaluation procedures, and (6) role of the 
instructor, will result in more effective and efficient instruction. Programming is funda- 
mental to the instructional system as the basic means of communications; programming 
techniques may be applied to written, oral, or visual presentations. Two courses 
presented by the instructional system over closed circuit television, basic Aerodynamics 
and basic Spanish and written programs in mathematics and chemistry have indicated 
a 25% saving in student time with equal student achievement as compared to con- 
ventially taught cadets. “Oral" programming, as used in the instructional system 
required about 10 hours of instructor preparation, per lesson; written programming 
required 30-50 hours of instructor preparation, per lesson. 



The Air Force Academy began investigating new 
teaching devices and techniques in November of 1960. 
Our initial investigation concentrated on closed circuit 
television, programmed instruction, and accelerated 
reading techniques. Inevitably, we were drawn into 
considering all facets of the instructional process. As 
a result of the promising conclusions from our early 
investigations a Directorate of Instructional Research 
composed of active faculty members, was formed in 
August, 1961. 

The Academy has evaluated new teaching devices 
and techniques, including programming, as they apply 
to an undergraduate academic institution. The military 
training aspect of the Academy’s program has not been 
considered up to this point although we recognize that 
profitable applications may be found. 

We have concluded that more effective and eflScient 
instruction will result from a system of instruction. The 
system used may vary from course to course but in all 
cases it is developed from a thorough analysis of: (1) 
course objectives, (2) learning processes, (3) media 
and materials, (4) methods of presentation, (5) evalu- 
ation procedures, and (6) role of the instructor. 

Programming is fundamental to the instructional 
system. It is the most effective and efficient communic- 
ation technique in the instructional process. The prin- 
ciples of programming apply equally to written, oral, 
or visual presentations. 

Two courses are presently taught at the Academy 
by means of the instructional system — ^Basic Aerody- 
namics to second class (junior) cadets and Basic Span- 
ish, a course in the base education program. 

Currently basic aerodynamics is being presented to 
two groups of 110 cadets each over our closed circuit 
television system. At the beginning of every lesson 
each cadet is given a handout similar to a linear pro- 
gram with approximately 30 “frames”. The instructor 
“programs” Ws oral and visual presentation of the 
lesson; this material along with the student handout 



forms a program of instruction. For instance, the in- 
structor may explain how pressure varies in the atmos- 
phere and show a short film clip on the subject. At 
preselected points the instructor will direct the students 
to read and respond to a specific frame on the hand- 
out. The students respond and immediately check the 
correctness of their answer against the printed answer. 
They inform the instructor if they have answered cor- 
rectly over a student-instructor communications system 
(Teletest). The instructor evaluates the collective re- 
sponse; if he is satisfied he continues with the instruc- 
tion program; if not, he may repeat, include other 
explanatory material (branch), or simply allow more 
time to cover each point. 

After the program of televised instruction is con- 
cluded, the student continues to the second part of the 
handout, also in the linear format, which is a pro- 
grammed review and reinforcement of the preceding 
instruction. The student completes this part at his 
own pace. An instructor is always available to answer 
questions during this portion of the lesson or repeat 
any portions of the preceding instruction. In a similar 
manner, a third part of problems or applications of the 
lesson material is subsequently completed. The student 
is permitted to leave after completing parts two and 
three. 

About 10 hours per lesson were required to prepare 
the student handout, the lesson materials, and the 
televised presentations. Similar lessons, prepared as 
written programmed instruction, required about 50 
hours of preparation per lesson. 

The Basic Spanish course is presented by television 
over the base community antenna system. The pro- 
grams may be received in over 1250 family quarters, 
airmen’s dormitories, the base hospital, and several 
other buildings served by the distribution system. Each 
participating student was sent a set of materials for 
every lesson. Each of the 56 lessons, consisting of 50 
frames each, is presented twice (at 1830 on the first 
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day and 1645 on the following day) before a live 
studio audience. The material is presented by a method 
of “oral” programming, similar to that used in aerody- 
namics. The instructor speaks the content of each 
new frame which may be a new word or an expression, 
offers any explanatory material, asks the class to re- 
peat what he has said, corrects the class response 
if necessary, shows the printed frame on the screen, 
and then requires the class tc write the frame in 
the space provided in the lesson materials. The pace 
of his presentation is dictated by the response of the 
class, which he can observe closely. After the conclu- 
sion of each 45 minute lesson, the student is advised 
to complete the lesson drill or review which is also part 
of the lesson material. A programmed review follows 
every eighth lesson and a programmed test is given 
during the next period. Each student corrects his own 
test according to the instructor direction, scores it, and 
mails it to the instructor for entry on the student’s 
record. 

One instructor prepares the materials, supervises the 
clerical support of over 500 students, and presents the 
lesson over television. Approximately 10 hours per 
lesson are required. The course includes a vocabulary 
in excess of 800 words and is equivalent to one year 
of high school Spanish. 

Both of these courses use the fundamentals of the 
instructional system; programmed presentations in con- 
jimction with student programs, student-instructor feed- 
back, branching and pacing based on the student re- 
sponse, and student paced drill and review. Both 
courses rely heavily on the student-instructor relation- 
ship and tend to maximize, rather than minimize, the 
role of the instructor. 

The course in aerodynamics was prevoiusly pre- 
sented to 44 cadets during the fall of ’63. The group 
achieved equal learning (as compared to cadets taught 
by conventional instruction) ; the cadets averaged about 
25% less time per lesson (as compared to convention- 
ally taught cadets). Standard examinations were given 
to each group. Scores in the Spanish examinations 
have averaged over 90% for all age groups, from 
3 years of age and up. 

The flexibility of the instructional system is illus- 



trated by the fact that it can be adapted to such diverse 
subjects as Spanish and aerodynamics. The techniques 
of presentation differ according to the nature of the les- 
son materials but the fundamentals upon which the 
techniques are based remain the same. 

We have also used conventionally written programs 
in two courses, for a 12 hour block of statistics included 
in our beginning mathematics course and for a 13-hour 
block in our be^nning chemistry course. Approxi- 
mately equal learning was achieved in each case as 
compared with conventionally taught classes with a sig- 
nificant time savings for the student. The written pro- 
grams were used as the sole means of instruction in 
both instances. We are convinced, however, that such 
programs can be integrated profitably into the .system 
of instruction and become an effective means of pre- 
senting certain types of basic materials. 

The latter two courses required 30-50 hours of in- 
structor time to prepare one class hour of instruction 
as opposed to 10 hours per lesson in aerodynamics and 
Spanish. 

We have concluded that programming can be very 
effective in higher education. It is an unfortunate fact 
that conventional programming is used sparingly, if at 
all on most campuses. The reasons for this, however, 
are fairly obvious; (1) written programs have been 
proposed by their advocates as primary vehicle of in- 
struction thus pre-empting the activity of the instructor 
in the learning process; (2) most instructors are re- 
luctant to accept the passive role thus created for them; 
and (3) conventional programming is complex in terms 
of manpower, laboriousness, time, expense, and in flexi- 
bility of the finished product. 

llie apparent solution is to enlarge the appreciation 
of programming as a means of basic communication, 
simplify the development of programs, introduce flexi- 
bility and adaptability into their use, and clearly dem- 
onstrate to the instructor how he may use programs 
profitably to his satisfaction. 

We believe the instructional system is a solution 
which can be applied to any academy situation within 
a wide variety of physical resources. The initiative and 
creativity of the instructor is the key to success rather 
than expensive or elaborate equipment. 
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PROGRAMMED INSTRUCTION AND ON-THE-JOB TRAINING 

W. D. Ceely, Colonel, USAF and Jimmie Green> Lt Colonel, USAF 

Colonel Ceely and Lt. Col. Green of the Air Training Command describe the Air 
Force’s development of programmed instruction packages for on-the-|ob training (OJT). 
The rapid technological advances in Air Force weapons and equipment demand con- 
comitant advances in training. This report outlines the required OJT concepts, the 
criteria established to implement them, and the techniques used to evaluate the pro- 
grams. Programmed instruction is treated as a key component in an extensive 
geographyicaliy dispersed training system. 



As recently as 15 years ago on-the-job tr ainin g 
played a relatively minor role in the Air Force training 
system. Most of the weapons and equipment of that 
day were basically the same as those used during World 
War II, except for a growing number of jet-powered 
aircraft. Training of maintenance and operations per- 
sonnel presented no special problems. 

During the 1950’s, changes occurred with ever- 
increasing frequency. Our entire force of combat air- 
craft was converted from conventional power-plants to 
jets. A sophisticated and extremely complex Air De- 
fense system was developed. The groundwork was laid 
for the space program which got underway in the late 
1950’s. Throughout the Air Force, complex machines 
came into use to perform many tasks. 

During this technical revolution, on-the-job training 
(OJT) failed to keep pace. The basic OJT method 
dates back to the Middle Ages. There is no fundamen- 
tal difference between the ancient guild system and the 
method the 20th century computer maintenance tech- 
nician employs to train his apprentice. During the 
Middle Ages, the apprentice was trained slowly and 
carefully under the close supervision of the journeyman, 
just as he is today. 

By 1960 it became apparent that Air Force opera- 
tional units were experiencing more and more difficulty 
in developing skilled airmen to operate and maintain 
the complex new weapons and equipment. Several fac- 
tors hampered on-the-job training to the extent that it 
became a major problem. Frequently, the only equip- 
ment available for training was fully operational. This 
placed serious time limitations on the leam-by-doing 
phase of OJT. The knowledges required for satisfac- 
tory performance became more extensive. Fewer opera- 
tional units had the required classroom space, and even 
fewer had adequate numbers of qualified instructors. 
The training publications designed during the 1950’s 
were intended for joint use by supervisor and trainee, 
but supervisors no longer had the time to provide the 
personal assistance needed for effective use of these 
materials. The problem was further aggravated by the 
fact that it became increasingly difficult to keep the 
training materials technically current. 



To see the problem in its proper perspective, we 
must realize that every year the Air Force trains about 
200,000 airmen in more than 1,000 different specialr 
ties by means of OJT. In October 1960, several studies 
were initiated in an attempt to find a more effective 
method for conducting OJT. The principal finding of 
the studies was that the Air Force could no longer 
afford to use traditional methods of conducting OJT. 

Most of the training problems in the operational 
units were in the area of knowledges rather than skills; 
There were only two apparent solutions to the problem 
of knowledge training. All the knowledge could be 
provided in formal technical schools, or the trainee 
could use self-study publications during his off-duty 
time. The high cost of formal training, plus the neces- 
sity for removing the trainee from his job, made it 
impractical to expand formal training. 

The outcome of the studies was the development of 
a new concept of OJT. These were the basic elements: 

1. The primary responsibility of Air Training Com- 
mand (ATC) with respect to the Air Force OJT pro- 
gram will be to provide self-study publications which 
will provide all the knowledges required for airmen 
upgrading in each specialty. These will be designated 
Career Development Courses. 

2. ATC’s secondary responsibility will be to assist 
the Major Air Commands in developing job proficiency 
guides. These guides will contain task inventories and 
reading references. In some critical skills, test ques- 
tions and work exercises will be added. The guides will 
be cellular in construction, so that each command and 
its subordinate units can provide supplements as re- 
quired. 

ATC’s new role in the Air Force OJT system re- 
quires a careful approach to the problem of developing 
effective and inexpensive self-study materials. These 
following criteria have been established: 

1. The subject matter will be limited to fundamen- 
tals and basic principles. References to specific items 
of equipment will be limited to illustrations of basic 
characteristics. 

2. The level of treatment of the subject matter will 
be directly related to the skill level of the trainee. 
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3. Each course will be self-contained. The trainee 
will be studying in his quarters and will not have access 
to other publications. 

4. All material in the courses wiU be unclassified to 
simplify handling problems. 

5. The courses will provide all knowledges required 
for upgrading in each specialty. End-of-course tests 
should be the qualifying examinations for upgrading, 
as Specialty Knowledge Tests are now. 

6. To avoid duplication, material common to two 
or more specialties should be contained in separate 
generalized courses. For example, a course will be 
required on the Air Force maintenance management 
system. This knowledge is required in several mainte- 
nance specialties. 

In 1961, the Air Force began a series of experiments 
in programmed instruction. It soon became apparent 
that this method of training had a direct application 
to the self-study courses to be developed by ATC. Pro- 
grammed instructional materials have two characteris- 
tics which make them uniquely suitable for use in 
Career Extension Courses: They are self-contained and 
they are designed specifically for self-study. 

In consideration of the advantages and disadvantages 
previously described, a project was established to pre- 
pare a programmed package on subjects common to all 
airman specialties. This package will provide training 
on Security, Ground Safety, the Air Force Publications 
System, Supply Procedures, Training, and Principles 
of Supervision. This package has 200,000 potential 
users each year. Airmen going direct from Basic Mili- 
tary Training to operational units will need all the 
material. Airmen who have completed technical school 
and are on OJT for progression to a higher skill level 
will need all or part of the material for review prior 
to taking the qualifying tests. 

This general package represents the largest known 
application of programmed instruction principles. It 
will be followed by other courses on subjects common 
to a particular group of specialties. As an example, 
airmen in all missile specialties will need a course on 
guided missile fundamentals. 

The size and scope of the project present some spe- 
cial problems in training management. The diversity 
of subject matter in the package require^, that parts of 
the project be developed in several Technical Training 
Centers, each of which is responsible for training in 



different subject areas. A central control agency was 
established to coordinate the work of the participating 
centers, and to exercise quality control over all phases 
of the work. Lackland Military Training Center was 
selected to perform these functions. 

The extremely large volume of printed matter dic- 
tated a requirement for maximum standardization of 
format. Experimentation was encouraged to determine 
the best possible format within the established limits 
of time and funds. 

Evaluation will be the most difficult problem. Evalu- 
ations of each volume must be based on 3 different 
measurements: 

1. First, the programmed material must be meas- 
ured against conventional material prepared to the same 
set of objectives. This requires development of the 
same course in conventional format. Since all the 
conventional material is already in existence in other 
publications, no significant increase in workload is an- 
ticipated in developing the conventional course. 

2. Next, local tryouts must be conducted to evaluate 
the suitability of the programmed material for wide- 
scale use in the field. While this appears to be a rela- 
tively simple matter, the large volume of material to 
be used in formal OJT programs requires that this 
phase of evaluation be as error-free as possible. 

3. Finally, field evaluations must be conducted to 
compare the programmed and conventional materials 
and to determine the effectiveness of the programmed 
materials. For the latter purpose, a system of continu- 
ing evaluation must be established. 

At this time, the Career Development Course project 
is slightly ahead of schedule. Criterion objectives have 
been submitted to the prime center for review and 
approval, and the follow-on work is in progress. We 
still have little to report in the way of significant results, 
but we expect to learn a great deal about large-scale 
programming from this project. The greatest deterrent 
to success is perhaps the cautious attitude we have 
developed toward management of the project. The great 
potential impact of our mistakes has tempered our 
experimental zeal. We are encouraged, however, by 
the thought that we may make a significant contribution 
by easing the training burden in every Air Force organi- 
zation and at the same time giving the trainee a publi- 
cation which is both easier to use and much more 
effective. 



RESEARCH ON AUTOMATED TRAINING AT ELECTRONICS SYSTEMS DIVISION 

Sylvia R. Mayer 

Will programmed instruction hold its Initial effectiveness and motivational charac- 
teristics over a period as long as two years? Can the operating equipment of Infor- 
mation Systems be used to automatically train its own operators? Can programmed 
instruction effectively train management level personnel In decision-making? Answers 
to these and like questions are being sought by the Air Force Systems Command and 
progress is reported by Sylvia R. Mayer. 



Air Force Electronic Systems Division produces the 
gigantic Information Systems for Air Force Command 
and Control Functions. Included in this new class of 
systems are such familiar computerized wonders as 
these: SAGE, SAC Control System, NORAD Combat 
Operations Center, Air Force Control System, Space 
Track and many less familiar. 

These Information Systems differ in their military 
missions such as command, air defense, aircraft or 
space vehicle control and military intelligence produc- 
tion. However, the equipment, operations and be- 
havioral requirements of personnel are strikingly similar. 
The raison d’etre of all Information Systems is the 
same. They are designed to aid and extend the user’s 
intellectual functions. 

A large part of the training research for these sys- 
tems is being cast in the framework of automated train- 
ing and programmed instruction for on-the-job training. 
For this brief discussion I would like to sketch the four 
programmed instruction trends which have been emerg- 
ing from our research since its beginnings in 1957. 
These trends are concerned with: 

I. A new technology for implementation of pro- 
grammed instruction. 

II. Types of subject-matter unique to Information 
Systems. 

III. Programming techniques tailored to this subject- 
matter. 

IV. Scope of application. 

I. Technology of Implementation. 

A. The Teaching Machine as a Part-Task Trainer. 

In 1958 we initiated automated training in the SAGE 
System. The conviction that teaching machines would 
be useful for on-the-job training in SAGE operations 
was an outgrowth of several factors. First, an analysis 
of SAGE operator jobs revealed a large portion to be 
dependent on intellectual functions. These functions 
can be part-trained and practiced on teaching machines. 
(Mayer, 1959b, and 1960b) 

A second factor concerned the economics of the 
SAGE on-the-job training problem. Frequent changes 
in cpmputer-program procedures and equipment char- 
acterize Information Systems. Such system evolution 



results in heavy training and retraining requirements. 
Teaching machines could help meet these demands 
economically. (Mayer, 1959a) 

Two teaching machines were provided in two differ- 
ent SAGE sites. One was the Air Force’s experimental 
Subject-Matter Trainer. The other the Navy Autorater. 
Programs were developed and training personnel given 
guidance for development of additional programs. The 
programming technique was of the type introduced by 
Pressey. 

We were interested in several questions. Would the 
crews continue to use the machine over a long period 
of time, or was their initial interest due to its novelty? 
Could the technical content of SAGE jobs be analyzed 
into the formats permitted by the trainer design? Third, 
we wanted to see what kinds of administrative and 
procedural problems might arise such as scheduling, 
developing materials, maintaining the trainers, etc. It 
seemed useful to determine if the apparent training 
value of the devices would outweigh the problems they 
introduced. (Mayer, 1961) 

The results were encouraging. On their own the 
training staffs worked out a procedure to capitalize on 
the natural rivalry between three operating crews. 
Scores from performance on the trainer were recorded 
and exhibited for each crew, rather than for individuals. 
Peer as well as supervisory pressure was brought to bear 
on each individual to achieve 100% scores on the 
teaching machine. 

In addition to fulfilling their primary instructional 
function these trainers also provided a secondary enter- 
tainment function. The obvious gaming features of 
these devices were readily recognized by the crews. 
Their entertainment value in conjunction with training 
is not to be minimiz ed in an environment such as SAGE 
where boredom and fatigue from homogeneous stimula- 
tion can be deadly. (Mayer, 1958) 

Records of usage \vere maintained over two years. 
Usage continued at a steady rate over this period. Only 
two real problems were identified. One, the machines 
could not stand up under the heavy usage. Mainte- 
nance was and is a recurring problem. Two, the train- 
ing staff had difficulty keeping up with the continuous 
demand for new training materials for the machines. 
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The conclusion reached was that automated training 
with teaching machines of this type is useful for oper- 
ations similar to SAGE. The major lesson learned was 
that something must be done to automate development 
of programs, if the full potential of automated on-the- 
job training is to be realized. (Mayer, 1960a) 

B. The Teaching Machine as a Whole-Task Trainer. 

The success of this first simple feasibility study led to 
a more ambitious study. The first study was concerned 
with automated part-task training — such as learning 
symbology, procedures, and rules, which were more or 
less separate entities. The second study was an attempt 
at automated whole-task training to learn both the 
background knowledge and operation of the system. 
An effort was made to program the entire learning 
task by using a combination of a conventional teaching 
machine for program control and feedback, plus a 
SAGE console simulator, plus flash-cards. Programs 
and machines were developed under contract with 
American Institute for Research. We tested and evalu- 
ated these materials for on-the-job primary position 
training in one SAGE sector, and for cross-training of 
operators in another SAGE sector. Subjects achieved 
test scores equivalent to those of operators with several 
years experience. Thus, there was no question as to the 
efficiency of this method from the trainees’ point of 
view. However, from the trmning administrator’s view- 
point just keeping the finished programs, training dis- 
plays and simulator hardware features updated to 
match the. evolving operational system was a major 
problem, even during the six months of the evaluation. 
(Shettel, 1961) 

C. The Built-in Teaching Machine. 

From these two studies and another similarly con- 
cerned with , automated maintenance training, we con- 
cluded that research was required for improved imple- 
mentation for automated training and programmed 
instruction in Information Systems. Specifically, we 
moved away from the concept of using separate teach- 
ing machines or simulators with fixed programs. We 
have initiated a long-range research effort to design into 
otherwise conventional operational equipment the ca- 
pacity to automatically train its own operators. (Mayer, 
1962) 

Current work is seeking to determine and demon- 
strate through use of a model how the logical functions 
^d mechanisms of the operating task and associated 
program and equipment may overlap with the functions 
and. mechanisms of the training program and equip- 
ment. We are trying to define how to exploit this func- 
tional overlap in order that the operational equipment 
may be used at one time to train operators while at the 
same time it is used by other operators to perform the 
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piimaiy task. Information Systems all have enormous 
computing capacity which will be exploited to provide 
computer-directed training. This long-range research 
program requires the solution of many problems con- 
cerned with computer program decision structure, auto- 
mation of program design, new computer input-output 
devices for the training function, etc. Contractual sup- 
port is being provided by Massachusetts Institute of 
Technology and Biodynamics, Inc. 

D. Computer-directed Training in System 473L. 

We are now at the beginning of a new system study 
in which we are trying out on a small scale the concept 
of using the operational system to train its own oper- 
ators. This work is on the new Air Force Control Sys- 
tem now in development. We are planning automated 
training to be presented in part by programmed text- 
book and in part by the system computer through the 
system consoles. The training task is to learn to com- 
municate with the system computer in its special query 
language in order to retrieve stored information from 
the system data files. 

The computer-directed part of the programmed in- 
struction will be stored in the system computer, and be 
presented at the system consoles. The trainee will re- 
spond on the console message composers. The com- 
puter-directed portion of training will be automatically 
generated and scored. Contractual support is being 
provided by American Institute for Research. 

II. lype of Subject-Matter. 

A second major area of work concerns the adapta- 
tion of programmed instruction principles to a wide 
sample of the tasks, subject-matters, and skills required 
in Mormation Systems. 

In addition to the operator-skill training studies I 
have already noted, we have several projects completed 
and several underway exploring the use of programmed 
textbooks for trouble-shooting training of electronic 
maintenance technicians. (Mayer, 1963, and Weiss, 
1960) Contractual support has been provided by 
Matrix Corporation. 

We also are experimenting with the use of pro- 
grammed instruction for training management level per- 
sonnel in decision-making. A programmed course v/as 
recently developed and tested on the PERT System, the 
new management technique for program evaluation and 
review. Contractual support is being provided by Ray- 
theon Company. This project demonstrated the feasibil- 
ity of using the programmed instruction method for 
on-the-job management training. We are now working 
on the plans for a follow-on managers’ course called 
PERT/COST, which is concerned with fund allocation 
techniques. This is being planned for use throughout 
the Department of Defense. 
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III. Programming Techniques. 

These adaptations of programmed instruction have 
generated the need for programming techniques which 
differ in some ways from the more familiar program- 
ming for academic subjects. This fact ties in with our 
third area of interest: programming techniques. 

A. The Model. 

The underlying instructional model being automated 
is different, especially in operator training. Instead of 
the Socratic model of student-tutor interaction, we are 
automating the master-apprentice interaction model — 
referred to in military settings as “over-the-shoulder 
training.” In addition to the more conventional pro- 
gramming methods, we need programs in which the 
master demonstrates, coaches, guides, points, prompts, 
redirects the apprentice’s performance or requires the 
apprentice to copy, imitate, match, fill-in, start or finish 
the master sample. 

B. Visual Aids Methodology. 

In addition, our programming leans heavily on visual 
aids methodology. Whenever possible, pictures, graphs, 
diagrams, pointers, signals, symbols replace words. 

C. Testing. 

Short test sequences are integrated within all our 
programs. Pre- and post-test procedures are treated as 
part of our programmed instruction packages. 

D. Basic Research. 

To fill our long range needs, we are also sponsoring 
research on a new approach to programming. Con- 
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tractual support is being provided by Harvard Uni- 
versity. This work holds promise for development of 
new and highly simplified techniques of programming. 

rV. Scope of Application. 

Our programmed instruction research is ranging 
across various traditional training levels: position train- 
ing, proficiency maintenance, cross training, retraining. 
It is also ranging across various personnel levels: from 
operator-technician level, to system commander, to the 
highest-level managers. The design concept of many 
of these Information Systems implies that the top-level 
analysts will personally interact with the equipment. 
Thus in our work we see a trend towards demands 
for programmed instruction and automated training 
techniques suited to highly sophisticated operators. We 
also see an increasing requirement for adapting pro- 
grammed instruction techniques to advanced pro- 
ficiency maintenance training. 

In summary programmed instruction research for 
Information Systems is leading to the following: 

I. An advanced technology for implementing pro- 
grammed instruction. 

II. Application of programmed instruction to new 
subject matters and skills. 

III. Development of new programming techniques 
for these subject matters and skills. 

IV. Use of programmed instruction with highly 
sophisticated learners, and for advanced train- 
ing as well as for initial learning. 



VIII The Health Sciences 

PROGRAMMED INSTRUCTION IN MEDICAL EDUCATION 

M. S. White, Major General, USAF 

General White quotes from a report of the American Association of Medical Col- 
leges which suggest that in order to meet the growing national need for trained 
physicians, medical schools must change their objectives from the acquisition of 
knowledge to the selection, organization, and evoluation of new knowledge as a 
means of continuing post degree education. Programmed instruction is a particularly 
promising educational innovation that can help both medical students and practicing 
physicians meet this objective. 



As required for all new modalities introduced in 
education, the first flush of enthusiasm for programmed 
instruction must be tempered with a careful analysis 
of its potential value and limitations. A study of the 
first few exploratory uses of this innovation in the 
medical field, and a survey of its application in other 
areas, appear to establish a definite place for pro- 
grammed mstruction in medical education. 

The problems facing medical education now and in 
the future are receiving increasing attenion from not 
only the medical profession itself, but also from the 
national government. Various studies of the number 
of physicians required for the predicted population 
growth within this country resulted in recommendations 
for the construction of more medical schools. The need 
for 10,000 physician graduates per year has been 
projected compared to the present average of 7,000 
who are the product of approximately 85 medical 
schools. Associated with this demand for physicians 
is the requirement for additional ancillary supporting 
personnel required in the health fields. 

However paramount the problem of medical edu- 
cation is that concerned with staying current with 
the increasing scope of knowledge made available 
through today’s rapid technological advances. This 
applies in the undergraduate field where the limited 
four years of study presents the dijfficult task to the 
embryotic physician of absorbing all the world’s medi- 
cal knowledge. It applies particularly in the graduate 
field or lifetime learning period required of all physi- 
cians, if they are to provide the patients with the best 
of preventive and therapeutic care. 

In medical school the student must be exposed to a 
solid fundamental education. This long basic period 
required for the education of a doctor has received con- 
tinuous study and challenge. Its further lengthening to 
absorb the increased knowledge available in all special- 
ties of medicine, would decrease still more the produc- 
tive years available for a physician to render services. 
Therefore, as one of the objectives of undergraduate 
medical education, the American Association of Medi- 



cal Colleges has enunciated that the student must estab- 
lish essential habits of continuing self-education which 
will be used when he is no longer confined within in- 
stitutional walls. With great research advances produc- 
ing reams of additional scientific information each day, 
there is an essential need for a rapid means of making 
this material available to the practitioner for the good 
of his patients. 

In recognition of this problem, a joint study com- 
mittee in continuing medical education sponsored by 
many of the leading medical associations of the coun- 
try, under the aegis of the American Association of 
Medical Colleges, recently made a penetrating analysis 
and proposed a solution. In essence, the solution calls 
for a nationwide university without walls, that will 
provide educational opportunities for physicians on an 
accessible, convenient and individual basis. 

The report states: “To accomplish this, education for 
medicine must move from objectives placing such great 
emphasis upon the acquisition of information toward 
objectives more effectively promoting the ability to 
select, organize, and evaluate information in a way that 
wiU stimulate, as well as satisfy curiosity and also 
foster the kind of critical perception which the judg- 
ments of a learned profession will increasingly require. 
In this shift of objectives the acquisition of information 
becomes a means to an end as well as an end in itself 
and the manner of learning as well as what is learned 
can receive the proper emphasis”. 

The report then states: “This reassessment of objec- 
tives with consequent changes in methods of teaching 
and learning must occur at all levels of physician edu- 
cation: the medical school, internship, residency, and 
professional practice”. 

It is in this context then that we can determine how 
programmed instruction can be used. Paramount must 
be kept in mind that a curriculum must be developed 
that will best meet the end objectives, regardless of the 
teaching methods used. A teaching medium must not 
dictate the curriculum. Thus, in many medical areas, in- 
struction cannot take place except through a preceptor 
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closely supervising a student’s handling of live pa- 
tients. In other areas such as the reports of latest 
medical research, mass teaching can be accomplished 
through the use of video tapes and instructional tele- 
vision. In areas of basic fundamental teaching and 
continuing education, it can be seen that programmed 
learning techniques could be particularly advantageous. 
Programmed instruction can provide for individual in- 
struction at places and times convenient to both the 
undergraduate and graduate physician. Programmed 
instruction can be pursued at a pace commensurate 
with the student’s ability. Programmed instruction can 
assure a minimum standard of academic performance, 
an assurance that the knowledge necessary in any par- 
ticular subject has been attained. As a result, the 
specter of facing the achievement testing procedures 
that have been customarily used should not preoccupy 
the attention of students, when demonstrating their 
qualifications in basic or advanced studies and special- 
ties. 

When used to supplement other educational tech- 
niques, the success of programmed instruction will de- 
pend on the development of effective programs. The 
skills of competent programmers must be allied with 
those of the scientific teacher. Each must be certain 
that the program has measurable objectives; that stu- 
dent gains are obtained based on criterion tests; that 
the program can be completed in a reasonable time; 
that retention of material persists over a realistic time 
period; that the program is geared to the ability and 
qualifications of the student; and that the program, 
before being adopted, was evaluated against alternate 
methods of instruction with comparative gains demon- 
strated. 

When these criteria are met, it is likely that im- 
proved quality and effectiveness of medical training will 
be accomplished. There can result a more widespread 
use of highly trained instructors, to meet the burgeon- 
ing needs of medical schools. Student time spent in 
acquiring basic factual knowledge can be reduced. 
There will also be lessened failures and repeats. In- 
structors will have more time to deal individually with 
students, and in laboratory and clinic, stimulate their 
intellectual curiosity. 

In accepting new techniques for imparting education, 
the zealous advocate may mistakenly feel that he has 
a new and easier system for presentation of the same 
material. This delusion must be dismissed, for an all 
together new approach to education should be attained. 
It is imperative that full advantage be made of these 
educational advances, whether programmed instruc- 
tion, instructional TV or others, by adapting to the 
new technique rather than a simple transfer of course 



material into the system. Most medical educational 
institutions now existent, designed about educational 
methods which have changed little in the past 50 years, 
permit little opportunity to start from scratch and ad- 
just their whole approach to instruction with these new 
methods. Evolutionary transition will be the rule in 
these instances as physical and other limitations are 
overcome. However, in the case of new institutions 
which are presently being planned, it is apparent that 
the educator can design his whole curricula and plant 
about the best distillation of all methods as they be- 
come proven, for the benefit of both students and 
instructors. Proper projection studios, classrooms, 
laboratories, individual study areas, libraries, clinics, 
operating rooms, consultation rooms, all can be de- 
signed for these new techniques at minimum cost com- 
pared to the rehabilitation of older institutions. Such 
a task presents a wonderful and formidable challenge. 

At the present time, the use of programmed instruc- 
tion in medical education has been initiated on a small 
and experimental scale in some medical schools, in 
military technical schools for training of para-medical 
personnel, in pharmaceutical companies for training of 
sales representatives who deal directly with physicians, 
and in post-graduate o^ continuing courses for physi- 
cians. 

Preliminary evaluation of results has indicated the 
effectiveness of this form of instruction. At Dartmouth 
Medical School in Hanover, New Hampshire, Drs. 
Green, Weiss and Nice are pioneering in research on 
the applicability of programmed instruction in a medi- 
cal school. Courses developed to date have been from 
a research point of view only, in order to evaluate the 
effectiveness of programmed learning at the graduate 
level. The courses tested have been parasitology, 
biochemistry, a review of the autonomic nervous system, 
in relation to the pharmacology course, gross anatomy 
and neuroanatomy. 

From the original experimental course, the investi- 
gators have concluded that programmed instruction is 
an effective technique for teaching a laboratory course 
such as parasitology. Students achieved a significantly 
higher test performance in this course compared to 
their grades following conventional instruction. Also, 
less study effort was required by the more productive 
students to reach the same high levels of performance, 
while the less productive students required less study 
time and achieved higher scores. 

At the University of Illinois College of Medicine, in 
Chicago, an experimental study is being made with the 
introduction of courses in body fluid metabolism, bio- 
chemistery, physiology and internal medicine. Simi- 
larly, experimental programs are being introduced at 
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the Western Reserve School of Medicine in Cleveland 
in hematology; at Columbia University College of 
Physicians and Surgeons in New York City in urology; 
and in the New York Hospital, Cornell Medical School 
on the thyroid gland. 

Opportunities appear legion for the application of 
this technique throughout other medical courses. Better 
utilization of a student’s study time can be made and 
he can obtain the background knowledge and freedom 
to profit from bedside and laboratory discussions with 
instructors. 

Medical courses developed for training of profes- 
sional sales representatives of drug companies have as 
their objective the attainment by the representative of a 
thorough understanding of the medical background, the 
product background, and the associated terminology 
necessary to discuss the drugs under consideration in- 
telligently with physicians. Programmed texts for this 
use have been developed in bacteriology and antibio- 
tics, cortico-steriod pharmacology, glucose metabolism 
and insulin, female endocrinology, cardiovascular phy- 
siology and diuretics, fungus diseases and griseofulvin. 
These courses vary in study time from one and one- 
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half hours to fifteen hours, with most requiring eight 
or nine hoars. 

In continuing education an example of programmed 
instruction’s potential contributions can be seen in the 
course on allergy and hypersensitivity made available 
in June, 1963, to all physicians without cost. Tne course 
consists of 361 frames and requires three to five hours 
to complete. The sponsoring company throughout the 
year, will distribute other short post-graduate courses, 
prepared by competent programmers working with sub- 
ject matter specialists from leading medical schools. 

Thus, we see the awakening of interests in the po- 
tentials of programmed instruction throughout all eche- 
lons of medical education. 

Preliminary experimental evidence indicates that 
there is opportunity for the adoption of this method of 
instruction with other new educational techniques in 
many segments of medical curricula. 

As continued experimental studies are made and 
future refinements develop, we will eventually see pro- 
grammed instruction fit into its proper niche for maxi- 
mum contribution to education in the health profes- 
sions. 



153 



Trends In Programmed Insfrueflon 



PROGRAMMED INSTRUCTION APPLIED TO SELECTED ASPECTS OF DENTAL EDUCATION 

Dc!e W. Podshadiey 

Date Podshadiey reports two studies which demonstrate autoinstructional methods 
can be used to teach basic dental science and public health materials to dental 
students. Part of the dental science material was revised into narrative form and used 
effectively as a refresher for practicing dentists. He concludes that wider use of 
programmed instruction could lead to greater instructor efficiency and curriculum 
flexibility. 



The problems of particular concern in the dental 
educational process today are not unlike those in the 
related health professions. At the undergraduate level, 
the dental profession shares with the other health 
sciences the task of incorporating the many technologi- 
cal and biological advances into an already crowded 
curriculum. It seeks, as they do, a curriculum which 
reflects social as well as scientific progress, and instruc- 
tion which utilizes to ihe fullest the findings of educa- 
tional research. And dentistry, perhaps even more than 
the other professions, is hampered in its efforts by a 
shortage of teachers. 

Coupled with the diflSculties at the undergraduate 
level is the problem of maintaining knowledge and 
skills among practitioners. The methods relied on to 
a great degree such as society meetings, journals, in- 
formal contacts, are limited in effectiveness and might 
be considered haphazard educational experiences. Re- 
fresher courses given by the universities probably are 
the best means currently available. But a busy prac- 
titioner has little free time and may hesitate to spend 
this time and the necessary money for courses which 
may be applicable only to a small part of his practice. 
In addition, there are too few schools offering refresher 
training and these schools are often inaccessible to the 
practitioners who are least able to stay abreast of new 
developments by other means. 

With this latter problem in mind, we decided on an 
exploratory study of programmed instruction as a 
means of facilitating refresher education. If a substan- 
tial portion of postgraduate materials were amenable 
to programming, we could make a significant contri- 
bution to improving the continuing education process 
of dentists. 

Our first project was contracted to Stanford Research 
Institute in cooperation with the School of Dentistry of 
the College of Physicians and Surgeons of San Fran- 
cisco. The essence of ..;ie study was to determine the 
feasibility of self-instructional procedures in refresher 
courses by estimating the effects, measured in learner 
performance and attitudes and costs, in time, effort, 
and dollars. 

Ultimately a branching program, in book form, was 



developed on the periodontal membrane, a segment of 
oral histology basic to refresher courses in periodontics, 
a specialty of dentistry concerned with the supportive 
structures of the teeth. The subject area permitted the 
use of first-year dental students as test groups prior to 
modification of the material for evaluation with prac- 
ticing dentists. 

In designing the study, an attempt was made to com- 
pare programmed instruction with two other forms, a 
seminar and a programmed lecture. The latter was pre- 
pared and delivered by a faculty member who had 
taken an active part in advising on the construction of 
the program and the criterion test. The intention was 
that the lecture duplicate the programmed text. The re- 
sults of this trial indicated that the self-instructional 
materials were as effective in teaching the subject 
matter as the programmed lecture and somewhat more 
effective than the seminar. However, the seminar group 
was not exposed to the precise teaching objectives. 
There was no indication that individual differences be- 
tween students with respect to their scores on an 
intelligence test or to their grade-point averages in 
previous courses affected test scores to any degree. 
The major place where differences were noted was in 
the time required to complete the program, from thirty 
to ninety minutes. 

On completion of this phase of the study, the mate- 
rial was refined for use with practicing dentists and an 
additional program (on the cementum) was prepared. 
Again to provide some comparison of the programs 
with conventional methods of instruction, narrative ver- 
sions of the programs were written. These narratives 
were identical to the program with the exception that 
the correct answers to the questions were phrased as 
statements and the questions were omitted. 

A major finding in a comparison of the two methods 
with matched groups of dentists was that the pro- 
grammed materials and narratives were equally effec- 
tive, with the narratives requiring less time for comple- 
tion. An assumption is that the narrative was sufficient 
to reinforce previous learning since aU the participants 
had been exposed to the material at some time in their 
professional careers. Of course, an unanswered question 



154 



is whether a nanative of equal effectiveness could have 
been prepared without the process of progr ammin g. 

The readers of both the- programs and the narratives 
thought the materials compared favorably with conven- 
tional textbooks although neither method was thought 
to be superior to a lecture. Of concern, in this regard, 
is the problem of programming material adaptable to 
individuals with various levels of knowledge in the sub- 
ject matter. In this instance, some of the participants 
probably were bored hj content which they either knew 
or recalled rapidly. Th importance, with conventional 
programming methods, of developing a program for a 
specific group with similar knowledge or no knowledge 
in the subject cannot be over-emphasized. However, 
whether a comparison of materials new to the partici- 
pants would have resulted in different reactions is un- 
known. 

With the project in oral histology underway, we 
decided to try programming an entirely different sub- 
ject, that of dental public health, which is tied more 
closely with philosophy and attitudes than with facts. 
Our primary interest was in methods which would 
conserve the time of the instructor, allow flexibility in 
relation to the students’ time and, in particular, assure 
that a student had been subjected to a pre-determined 
body of public health content. 

This program, also in a branching format and pre- 
pared by an experienced programmer, requires from 
one and a half to three hours for completion. Evalua- 
tion of the unit, entitled. Introduction to Dented Public 
Hecdth, has taken place in five schools, using either a 
“scrambled book” or a multiple-choice microfilm de- 
vice. Although analyses of the data are not complete. 



Dale W. Podshadley 

there is no doubt but that the program is effective. 
The reaction of the students, both dental and dental 
hygiene, has been more favorable than expected, with 
ninety per cent indicating a preference for additional 
programmed instruction. In addition, by far the major- 
ity of students consider the programmed material to be 
more effective than either conventional textbooks or 
lectures. 

On the basis of our experience, much of the lecture 
material in dental education (and undoubtedly this 
applies to the related professions), coidd be effectively 
and efficiently taught by self-instructional methods. The 
fundamentals in the basic sciences, where the laboratory 
sessions could be designed for reinforcement, seem par- 
ticularly amenable. Similarly, the fundamentals in the 
clinical sciences and many parts of the subjects rele- 
gated to the clinical years could be programmed. Per- 
haps the lecture time saved would provide the cur- 
riculum flexibility needed and the efficiency of instruc- 
tion would permit the addition of new courses or the 
expansion of existing ones. 

The surface has barely been touched in exploring 
the potential of programmed instruction, particularly 
the value of automated or machine-controlled teaching. 
Self-instructional procedures have a place in dental 
education now and will have a greater place as more 
efficient methods are developed. We are using the con- 
ventional methods in developing additional materials, 
but we also are planning more intensive studies, par- 
ticularly in relation to new methods. 

The experience of programming is invaluable if never 
carried beyond the development of precise instructional 
objectives. Perhaps this is where the major benefit ulti- 
mately will occur. 
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PROGRAMMED INSTRUCTION IN NURSING EDUCATION 

I • 

Evelyn Revels 

Evelyn Revels is active in the evaluation of nursing curricula for the state of Texas. 
She surveyed a nationwide sample of nurse educators to gather information on their 
interest in, and use of, programmed instruction. Her respondents expressed a desire 
for programs in a wide variety of subjects. But because of lack of knowledge or 
available material, relatively few of them are currently using programmed instruction. 



Nursing, in common with many professions, suffers 
from a great shortage of both teachers and practi- 
tioners. My own limited experience with programming 
has convinced me of its usefulness, and of its vast pos- 
sibilities as an effective aid in teaching some parts of 
nursing courses. I believe that it is vital to the future 
of nursing education that programmed instruction be 
developed to the most effective extent possible, and as 
soon as possible. 

However, before recommending nursing courses, or 
parts of courses, for programming, I felt it necessary 
to obtain specific information from the facilities of a 
nation-wide sampling of schools of nursing, so as to 
ascertain what is now being done in programmed in- 
struction, and also, what courses might profitably be 
programmed, in the opinion of nurse educators. 

Questionnaires were sent to 112 of the 1,084 schools 
of nursing in the nation, or one-tenth of the total. Ques- 
tionnaires were sent to every state except Alaska, which 
has no school of nursing, and to the District of Colum- 
bia and Puerto Rico. Every type of nursing curriculum 
was represented in the survey. 

About four-fifths, or 90 of the questionnaires were 
sent to the college-controlled programs, baccalaureate 
and associated degree granting institutions, the other 
one-fifth of the questionnaires were sent to diploma 
schools of nursing. The reason for the heavy weighting 
in favor of the colleges is that on a national basis the 
collegiate instructors are reputed to be the leaders in 
developing new and more effective methods of teaching. 
Of course, many diploma schools are very progressive 
in their methods of teaching. In addition, these schools 
continue to educate the majority of the nation’s nurses; 
therefore, it seemed desirable to obtain a sampling from 
these schools, as well. 

The response was excellent to the questionnaire, with 
85 (77%) of the questionnaires returned. Only five 
respondents (6% ) were using any kind of programmed 
materials, and of the 80 (94% ) not using programmed 
instruction, 25 (31%) stated that no appropriate ma- 
terial is available in nursing. Twenty-eight respondents 
(34%) indicated that they are either not informed 
about programmed instruction, or that their knowledge 
is limited. Eleven (14%) said that they are preparing 
their own material, or are planning to prepare their 



own, but do not yet have it ready. Instructors in six 
schools (7%) are studying programmed instruction 
and the materials available. Only three respondents 
(4%) questioned the value of the material available, 
and two respondents (2%) questioned the cost. 

Many of the respondents indicated that they would 
like to use programmed material for all of the courses 
listed. A breakdown of their preference is as foUows: 



Dosage and solutions 70% 

Medical vocabulary 55% 

Pharmacology 53% 

Signs, symptoms & pathology of clinical 

nursing courses 47% 

Anatomy and Physiology 37% 



In addition, faculty members listed the following 
courses as being appropriate for programming, and 
stated that they would like to have these courses pro- 
grammed; Skills in Nursing; History of Nursing; Chem- 
istry; Communications Skills; Asepsis in Operating 
Room; Legal Aspects of Nursing; Microbiology; Com- 
municable Disease Nursing (part of course); Descrip- 
tive Psychiatry; Nutrition; Leadership Skills (part of 
course in unit management) . 

Of the five respondents who are using programmed 
material, three schools prepared the material themselves 
for the following courses: (1) operating room nursing 
(part of the course), (2) maternal and child health 
nursing (part of the course), and (3) Anatomy and 
physiology and Medical-surgical nursing (parts of both 
courses) . 

The other two respondents who are using pro- 
grammed instruction materials have courses published 
by a commercial group. The courses are “Teaching 
the Diabetic Self-Care”, and “Asepsis Techniques”. 

Of the five schools using programmed instruction 
materials three appraised them as very valuable, one as 
valuable, and the other school said they are unable to 
evaluate the materials yet, since the}^ have just started 
using them. 

Based on the evidence gained in this survey, very 
little has been done, or is being done at present in devel- 
oping programmed instruction materials for nursing 
education, but a great deal of interest and enthusiasm 
concerning this field was expressed by nursing faculty 
members. 
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In the light of information gained from this nation- 
wide sampling, the following recommendations would 
seem in order: (1) that nurse faculty members review 
programmed materials now in existence to determine 
possible use in nursing education, (2) that individual 
nurse faculty members analyze courses they teach to 
identify parts which might be more effectively presented 
through programming and that they try programming 
some parts of their own courses. The process of pro- 
gramming a course forces the teacher to think more 
deeply about her course material, and she is a better 
instructor for it, and (3) that, where possible, persons 



Evelyn Revels 

trained in programming be used on a full-time basis to 
develop the programmed instruction materials that 
would be of value in nursing. These persons would 
need to work closely with nurses who are subject-matter 
specialists. 

The teacher using programmed instruction would be 
free to devote more time to the truly creative part of 
her teaching duties; that of teaching nursing students, 
by precept and example, how to apply the necessary 
scientific facts and principles in such a way as to give 
tender, compassionate care, filled with human unaer- 
standing, to the suffering sick under their care. 



IX Instructional Programming 

INCREASING YHE PRODUCTIVITY OF PROGRAMMERS 

Albeit E. Hickey 

Albert Hickey emphasizes the point that programming is still primarily an intuitive 
art and as such is inefficient and expensive. The author suggests concentration on 
development of better methods of selecting programmers, and on the development of 
algorithmic or recipe-like strategies for programmers. He also suggests the develop- 
ment of a rigorous and specific language structure for adequate application of the 
algorithmic programming system. 



As Donald Cook has observed recently, Watson, 
forty years ago, outlined the philosophy of behaviorism. 
Then, ten years ago, Skinner brought behaviorism to 
bear on education through the example of his teaching 
machine. Skinner’s teaching machine is important in 
this sequence, not only because it closed a gap between 
experimental psychology and education, but because it 
showed how autoinstruction could reverse the trend 
toward mass instruction by restoring the one-to-one 
relationship between student and teacher. If anyone 
doubts the social significance of autoinstruction, he 
need only consider that, since Skinner proposed his 
machine ten short years ago, elementary and high 
school enrollment has increased by 50% and college 
enrollment has nearly doubled. 

We are agreed then that we must proceed as quickly 
as possible to exploit the full potential of autoinstruc- 
tion. Many people, more or less interested in education 
and training, are rushing to fill a vacuum. But it is not 
a vacuum that will be filled by mere numbers. In the 
ten years that have elapsed since the introduction of 
the Skinner machine, we have not grown very much in 
our understanding of programming for autoinstruction, 
not nearly enough to make effective use of all the 
would-be programmers. 

This observation on the state of the art in program- 
ming should not be misconstrued as a statement that 
good programs do not exist, or that all the programs to 
be published will not be good ones. It is true, however, 
that all programs are being developed intuitively and 
then tested, a trial-and-error process that is much too 
inefficient to meet the great demand for programmed 
materials. 

The efficiency of programming affects all of us in 
programmed instruction. For example, as a developer 
and publisher of programmed materials, I cannot safely 
expand my company without an increase in the produc- 
tivity of programmers. The efficiency of the autoin- 
struction industry is also of concern to those of you 
who are here because you envision educational pro- 
gramming as a career. In the present circumstances 
many of you will experience frustration and disillusion- 



ment. The industry doesn’t know how to reliably select 
programmers, except by a protracted trial and error 
method that often leads to frustration on both sides. 
And once we select you, for better or worse, we don’t 
know how to instruct you so that we can be assured of 
a first class product with a reasonable expenditure of 
time and money. Finally, those of you who are here as 
potential users of programs will not be able to afford 
programs developed intuitively by only a few g^fted 
programmers. 

But programmed instruction can’t remain a cottage 
industry forever. What will be done to increase its pro- 
ductivity? There are at least three things I expect to 
happen. 

1. Sharpen our ability to select programmers. 

2. Improve our ability to give programmers frame- 
by-frame guidance through the development of some 
basic algorithmic models, and 

3. Develop an algorithmic or procedure-oriented 
language to aid in the analysis of the educational proc- 
ess and assist in communication between members of 
the programming team. 

First, in regard to selection, with the assistance of 
the Liberty Mutual Insurance Company, one of our 
clients, our company is seeking to establish norms for 
programmers on such standard selection devices as the 
Kuder Preference Test and the Thurstone Test of Men- 
tal Alertness. These data will accumulate very slowly, 
however, even in a business the size of an insurance 
company. I would suggest that the National Society 
or the Air Force, which seem to be largely congruent 
here in San Antonio, could help a great deal in this 
effort since they each, or both, represent the largest 
sample of programmers whose work can be judged. 
Of course, judging these programmed materials will be 
a problem itself. Lewis Eigen, executive vice president 
of the Center for Programmed Instruction, recently 
admitted that after writing and reading hundreds of 
thousands of frames he couldn’t tell a good one from a 
bad one. Fortunately, I think he was overstating his 
case. 



Trends in Programmed instruction 

A second requirement is for algorithms that are more 
detailed, explicit and testable than the gentle admoni- 
tions we now follow called RULEG, EGRULE, and 
“guided discovery”. Please don’t misunderstand me, 
these are important formalisms, but they do not yet 
contribute to productivity. 

Perhaps it would not be out of place to explain what 
we mean by an algorithm in learning. The range of 
learned behavior extends from, say, the eyeblink at one 
end of a continuum to problem solving and conceptu- 
ally mediated behavior at the other. Let us concentrate 
on the more sophisticated end. Problem solving and 
conceptual behavior are not mutually exclusive. To 
establish a concept it is often desirable to perform the 
operations that define the concept and in many cases 
the operations to be performed to solve a problem. 
Thus the teacher or programmer leads the student 
through a sequence of operations or responses. The 
operations are performed in a linear sequence because 
the student can perform, or at least attend to, only one 
operation at a time. This does not, of course, preclude 
branching at criterion points. 

What is the best sequence of operations for teaching 
a given concept? Unfortunately, no books tells you 
how to proceed from the point where you agree on 
terminal behavior to a frame-by-frame plan. Bruner 
observes that students left to their own devices show 
little agreement on strategjf. Further, observation of 
teachers’ classroom behavior and the analysis of text- 
books suggest that teachers, as lecturers and authors, 
do not follow a consistent sequence of operations in 
concept formation. Teachers of economics, for exam- 
ple, disagree markedly on an optional teaching strategy. 
Approximately 60 per cent of the teachers follow a 
micro-macro sequence, while the other 40 per cent sail 
the opposite course. 

At present the concept formation strategies used in 
programmed instruction sequences, like those in text- 
books, are being intuitively contrived. The claim that 
programmed sequences are “logically structured” refers 
at best to a very informal and personal logic. The 
current “lo^c” is not the logic of formal analysis. As 
I say, the closest approach to a formalism is the 
RULEG principle of Evans, Glaser, and Homme. 
Gagne and Brown have suggested a deductive or induc- 
tive sequence called “guided discovery”. Myron Wool- 
man has proposed the use of a programming lattice. 

All these recommendations are, however, too general. 
We continue to rely upon programming artists to plan 
concept formation strategies. Many programming activ- 
ities have organized programmers in teams around 
individuals with a teacher’s insight or instinct. But 
this is an inadequate step at best. The key man him- 
self would be more valuable, and his communication 



with other programmers improved, if he could talk 
more rationally about concept formation strategies. 

There is at least one ray of hope in this cloudy pic- 
ture. That is that them is reliable connection between 
the logical structure of a subject matter and the most 
efficient concept formation sequence. Let me speculate 
on what that relationship might be. 

Programmed instruction sequences for concept for- 
mation build toward a “terminal frame”, which is 
usually a verbal or graphic definition of the concept. 
The definition is a collection of the properties or quali- 
ties of the concept and their relationships. 

For example, the concept of a trian^e is comprised 
of two properties: threeness and polygonism. But poly- 
gonism is only the specification of more general prop- 
erties. A polygon is a plane. A plane is a magnitude. 
A magnitude is a quantity. Therefore, a triangle is a 
quantity. This chain is called a Goclenian sorite. It 
progresses from the more to the less abstract. In the 
case of the triangle, we have two sorites, one for numer- 
osity and one for polygonal, w'hich converge on the 
definition of the concept “triangle”. 

The definition of a more complex concept, for ex- 
ample the consumption function in economics, repre- 
sents the convergence of at least two properties, income 
and consumption spending, each of which is itself a 
complex concept. Consumption spending is the con- 
vergence of consumer and spending, and spending is 
the convergence of exchange, and goods and services. 
This hierarchy of concepts can be illustrated with a 
“tree” diagram, and a rather bushy tree at that. 

Since instruction is a “single-thread” process, the 
complex n-dimensional concept must somehow be 
reorganized into a linear verbal learning sequence or 
algorithm. In this transformation, which property of 
concept triangle should be introduced first in the learn- 
ing sequence for concept triangle, “3-ness” or “poly- 
gonal”? The artful programmer might at first say “it 
doesn’t matter”. Later he will usually feel he should 
take up one property before the other. For example, 
some properties or subconcepts have more involved 
derivations than others. This may lead to their earlier 
(or later) treatment in the linear program. To generate 
an experiment, I would venture this hypothesis: 

The most efficient learning sequence is the one in which 
each succeeding element is a more general case of the pre- 
ceding more specific conceptual elment. 

So much for algorithms. The third requirement is to 
develop a more efficient language with which to deal 
with these problems of program oxganization. English 
is too unwieldy. We can expect therefore to evolve a 
special algorithmic language similar in certain ways to 
the algorithmic language used to give instructions to 
computers. After all, students learning to solve problems 
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share a common characteristic with computers: they 
both respond to step-by-step instructions. A major 
differences between computers and students is that 
computers function solely to give solutions, while stu- 
dents solve problems not just to get answers, but to 
learn how to solve other related problems. 

To give step-by-step instructions to their problem- 
solving computers, mathematicians began some time 
ago to devise special languages such as ALGOL, 
COBOL, FORTRAN, LISP, and JOVIAL. These 
languages range from hardware oriented languages 
suited to specific computers, to meta-languages with a 
great deal of generality. Since educational program- 
mers are not yet constrained by machines, they should 
be able to develop an educational language of some 
generality. Now you can’t make any progress in re- 
search unless you have an acronym that sounds like a 
name, or a name that sounds like an acronym, so let’s 
call our new educational language DIDAC. Like other 
algorithmic languages, DIDAC will have a number of 
symbols standing for relational operators, logical op- 
erators, sequential operators, arithmetic operators, and 
declarators. The symbols will be composed into state- 
ments, labels, blocks, procedures, and programs. 
DIDAC will have a number of extra symbols, however, 
since the human brain can perform more sophisticated 
operations than an electronic computer. The most 
difficult task will be to specify the syntax, the rules 
governing the combination of symbols for most efficient 
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learning. The sequences will probably not be the same 
sequences used just to get quick answers. Tlie inven- 
tors of DIDAC will suffer a handicap not shared by the 
originators of computer languages. Psychologists have 
only a very poor understanding of the operations which 
result in human learning, while engineers have always 
been able to teU computer programmers how electroniv": 
computers function. After all, they invented them. 

Unlike instructions to computers, which are carried 
out entirely by the machine, instructions in DIDAC 
will be carried out partly by the teaching machine and 
partly by the student himself. In fact, present pro- 
grammed instructions look a lot like the script for a 
dialogue between a teacher and a student. Students 
will not need to know DIDAC, however. It will be 
used only to program the actions of teaching machines. 
The output of the machine will be plain language, or 
graphs and pictures. Classroom teachers probably will 
not find it necessary to learn DIDAC either, at least for 
the present. The language will be used only by educa- 
tional programmers. And nothing should enhance the 
status of programming as a profession more than a 
private language. 

In summary, programming is presently a craft indus- 
try, but the tremendous demand for autoinstruction 
calls for greater production. To achieve this we need 
better criteria for the selection of programmers, more 
detailed programming algorithms, and an algorithmic 
language for educational programming. 



161 



USEFUL OBJECTIVES 



Jeriy Short 

Jerry Short reminds us that a programmer can get positive reinforcement for talking 
about behavioral objectives, but to date there does not appear to be a contingency 
that will reinforce people for actually publishing them. More importantly, there does 
not appear to be a technology for developing frames from even a well developed 
objective. Dr. Short suggests a method for bridging the gap. He also suggests a par- 
tial format for a manual to accompany any programmed materials. His format should 
give a reinforcing market advantage to the programmer who actually follows it and 
publishes the behavioral objectives. 



One of the primary targets of the revolution called 
programmed instruction was the moldy Bastille of edu- 
cational objectives. Inside this prison under the old 
regime, courses were described in vague terms. Stu- 
dents were expected to understand, appreciate, and 
know things, and what this understanding, appreciation, 
and knowledge looked like was seldom considered and 
never elaborated. 

Since programming came directly from experimental 
laboratories concerned with the behavior of organisms, 
it is not surprising that its advocates viewed education 
as a process of behaviorial change. A program was 
not designed to convey information. It was designed to 
produce a change in the behavior of the student. Its 
objectives were not to impart knowledge, but to alter 
the student’s response to his environment. Thus, it was 
argued, objectives of programmed instruction could 
best be described in terms of the stimulating environ- 
ment and the student’s behavior in its presence. 

This concept is summarized in Robert Mager’s “Pre- 
paring Objectives for Programmed Instruction” (Mager, 
1961). In that book, the right answer is the action 
word which describes what the student does, not 
what he knows; how he behaves, not what he under- 
stands. Dr. Mager’s recommendations to describe edu- 
cational objectives in terms of stimulating environ- 
ments, modifying conditions, student’s responses, and 
criteria of acceptable achievement have been widely 
adopted in training programmers. And this is as it 
should be, for no characteristic of programmed instruc- 
tion is more important than the concept of education as 
a process of behavioral change. The concept that edu- 
cational objectives should be stated in terms of (1) 
stimulus conditions that can be replicated and (2) ob- 
servable responses that have a high probability of 
occurring in the presence of these stimuli is relevant. 

Everyone appears to agree that behavioral objectives 
have joined the select company of “motherhood” and 
“country”. They are good things in and of themselves. 
They must be determined before a frame is put on 
paper. They must be published so that programs can 
be evaluated. They must be reported in research 
studies so that other experimenters can judge and repli- 



cate the studies. But, if such objectives exist, they are 
well guarded secrets. I have looked in vain through 
published programs, manuals, and reports to find de- 
tailed objectives describing the behavior change that 
is to occur as a function of the student’s taking the 
program. I smile every time I see that someone else 
has suggested that a teacher or supervisor should 
choose a program by “comparing his course objectives 
with the program’s objectives”. I can only think of the 
poor teacher or curriculum supervisor comparing two 
blank pages, one not listing his own educational objec- 
tives in behavioral terms and the other not listing the 
program’s. 

Programmers, myself included, frequently talk about 
objectives, but often neglect them when writing a pro- 
gram. One reason for this neglect is the lack of 
aversive stimuli whose removal might reinforce writing 
objectives. This is in contrast to the aversive stimulus 
of a deadline that keeps many programmers writing 
frames: the Sergeant must produce frames so that the 
Lieutenant can show the work to his Captain who can 
demonstrate to a Major that he and his staff should 
continue to exist. The same aversive control exists in 
the chain of command from a programmer to his com- 
pany’s publisher. But the emphasis is always on pub- 
lishable frames, never on the publishable objectives. In 
addition, frame writing behavior is often controlled not 
only by the withdrawal of aversive stimuli, but also by 
positive reinforcement from supervisors: “Ah, you’ve 
produced 50 frames this week. Good! Here’s an extra 
package of frame paper.” In contrast, the behavior of 
writing objectives is seldom followed by anything re- 
sembling reinforcement. In fact, punishment may well 
be the usual environmental event that follows the pres- 
entation of a well formed objective to an uninformed 
supervisor. 

It can also be argued that little would be gained at 
the present time from publishing objectives even if 
programmers were reinforced for producing them. The 
behavioral objectives of programmers may sound lim- 
ited and meager in comparison to the claims of other 
educational marketers. The very precision that means 
so much to the programmer may indicate a limited 



coverage to the curriculum supervisor. Because a 
mathematics program does not claim to teach under- 
standing of addition, appreciation of subtraction, and 
a love of number concepts, it may seem a bad buy. 
The idea that the claims of educational techniques 
should be substantiated is new. A few years ago, no 
one had seriously thought of returning textbooks or 
films to their manufacturers because 10% of the class 
using them made F’s and obviously did not learn to 
love number concepts or appreciate subtraction. The 
precision of program objectives is impressive only when 
the objectives are paired with student data illustrating 
achievement of the objectives. Until such data are 
available, behavioral objectives will sound somewhat 
limited when compared to the advertised objectives of 
other techniques. 

Apparently the writing of behavioral objectives is not 
often reinforced by money, publication, praise, or other 
secondary reinforcers. Nor is it necessarily reinforced 
by effective frames and programs. Although much has 
been said about preparing objectives, a large void still 
exists between this step ard writing a program. There 
is usually a missing chapter in books on programming 
because no one has yet described exactly how a be- 
havioral objective can be converted to a program 
sequence. And many programmers, even after devoting 
considerable time to specifying what they plan to teach, 
have difficulty determining how they will teach it. 
Therefore writing objectives in behavioral terms is often 
not even rewarded by the rapid development of an 
effective program. 

Before I change from critic to advocate and before 
I suggest how writing objectives can be made reward- 
ing, I must specify two crucial elements of a behavioral 
objective. For unless the objective specifies both a 
stimulus and a response, all my recommendations are 
useless. 

Table 1 lists three objectives that we will assume 
refer to the same behavior pattern. The first objective 
can be criticized because it does not tell the instructor 
what to teach or how he can observe if his teaching 
is successful. It is an unreliable objective because dif- 
ferent teachers or programmers would teach different 



Table 1 

A. The student v/ill understand the nature of clas- 
sified materials. 

B. The student will define classified materials. 

C. The student will define classified materials as 
documents that, if seen by unauthorized per- 
sonnel, could damage the security of the 
United States. 
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things and, if they observed achievement at all, would 
certainly measure it differently. 

The first objective in Table 1 cannot be made more 
useful by substituting an “action” word for the word 
“understand.” Thus Objective B “The student will 
define classified materials,” is little better than Objec- 
tive A. We may have reduced the variance in teacher 
behavior by substituting the word “define” for the word 
“understand,” but different teachers would still teach 
different things and measure success in different ways. 

We can improve the objective by describing the 
response in more detail. Objective C in Table 1 seems 
behaviorally stated. However, it does not specify a 
stimulus. It does not tell what environmental situations 
wiU start the student defining. It might be argued that 
Objective C implies a stimulus such as the question 
“What is classified material?” But this may not be the 
stimulus that the programmer actually uses. His frames 
may indicate that he is using the definitional phrase as 
the stimulus and calling for the response “classified.” 
Or he may be interested only m having the student 
recognize the response phrase and therefore writes 
multiple choice frames in which “classified material” 
and the phrase defining it are both stimuli and the stu- 
dent makes a matching response. At least three stimu- 
lus-response patterns, all representing the “behavior 
of definmg,” could be correctly derived from Objective 
C. They are listed in Table 2. 

Only if an objective is stated in terms of a stimulat- 
mg environment and an observable response as illus- 
trated in Table 2, will it reliably describe the behavior 
that is to be taught. And only if it is stated in this 
way, will it program easily. 

I would like to describe one method that I have 
used to bridge the gap between an objective and the 
program materials written to achieve the objective. I 
will only briefly describe it here. You must wait 
anxiously until later when the method is either aban- 
doned or worked out in detail. 

Since the answer the student will make m the pro- 
gram is ideally the response specified by the objective, 
we begin writing the program sequence by making an 
educated estimate of the number of times the student 
will have to make this response or parts of it. If these 
responses are listed on the back of frame cards, we have 
some frames that are one-third complete. The program 
responses are relevant and important because they are 
are the responses required by the objective. In the pro- 
gram, a frame should not simply evoke a response, but 
evoke it in the presence of stimuli that will eventually 
come to control it. Since these are the stimuli listed 
in the objective, they can be transferred to the other 
side of the frame cards. Sometimes the exact stimulus 
listed in the objectives can be used, but often new 
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Table 2 

Objective C: The student will define classified 
materials as documents that, if seen 
by unauthorized personnel, could 
damage the security of the United 
States. 

Stimulus-response patterns suggested by the ob- 
jective: 

Stimulus Response 

1. Define classified mate- Documents that, if seen 

rial by unauthorized per- 

sonnel, could damage 
the security of the 
United States. 

2. Documents that, if seen 
by unauthorized per- 
sonnel, could damage 

U.S. security are called Classified material. 



3. Documents that, if seen 
by unauthorized per- 
sonnel, could damage 
the security of the U.S. 
are called: 

a 

b 

c 

d. classified 

materials Matches stem to 

choice d. 



examples or generalizations are required. Certain 
stimuli that should also elicit this response and certain 
“S” delta conditions (stimuli to which the response 
should not occur) are also added. 

The objective has yielded a number of pieces of 
paper with desired responses on one side and appro- 
priate stimuli on the other. These are then given to a 
Subject Matter Expert to see if the stimuli will evoke 
the desired responses from him. Unless the stimuli 
are effective enough to evoke this behavior in an 
expert, there is no reason to expect them to reliably 
evoke the response in the trainee. The incomplete 
frames are revised on the basis of the Subject Matter 
Expert’s responses. They are then sequenced and cued 
to assure that students will make the correct response 
in the presence of the appropriate stimulus. Each part 
of the frame is clearly distinguished by its function: 
the cue is used to assure a correct response, the frame 



context represents the stimulus' pattern that will even- 
tually control the response, and the response is clearly 
relevant to the objective. In this method, the mechanics 
of determining responses, establishing stimulus control, 
and adding effective cues precede the drafting of an 
item into a program frame. It is a method which 
promises to reward writing objectives by clearly speci- 
fying how a program sequence can be produced from 
an objective. 

My second suggestion for improving the reinforcing 
contingencies affecting writing objectives depends on 
the recommendations of evaluation groups such as the 
Joint Committee on Programmed Instruction and 
Teaching Machines formed by AERA-APA-DAVI. If 
the committee recommends a manual for programs 
that emphasizes the behavioral statement of objectives, 
then professional approval and, to some extent, sales 
will be contingent on writing these objectives. A pro- 
gram manual that emphasizes behavioral objectives 
might take the form shown in Table 3. 



Table 3 

Suggested format for program manual: 

OBJECTIVE 12: 

Stimulus Response 

When the student is 
given (1) a political map 
marked in degrees of 
latitude and longitude 
and (2) a question ask- 
ing him to name the city 
at — ^latitude (N or S) 

and — ^longitude (E or W) he will name the cor- 
rect city 



Criterion test items for Objective 12: 


Stimulus 


Response 


% of sample 
answering item 
correctly 


Use map A. V/rite 
the name of the 
cities located at 
these coordinates. 
33° 56' S, 18° 25' 
E. 




Before After 
Program Program 


30° 3' N, 31° 21' 
E. 


Cape Town 
Cairo 


4% 95% 

2% 98% 
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Under each objective, the manual lists the criterion test 
items and the field test data that indicate student 
achievement of this objective. This three-part descrip- 
tion of a program in terms of the behavioral statement 
of the objective, the criterion test items measuring 
achievement of the objective, and the student data 
showing success on the test item, would present a clear 
picture of what the program was designed to do, what 
the student was actually asked to do, and how well the 
program had succeeded 

A Subject Matter Expert might be challenged to 
publish a book in the form of Table 3 that set forth 
the objectives of some course in his special field. The 
book would list all the important behavioral objectives 
and give examples of what he meant by these objec- 
tives, i.e., problems which the student should be able to 
solve, sentences or essays which he should be able to 
write, etc., when he completed a course based on these 
objectives. Such a book would be controversial, in 
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itself a reinforcing event to any Subject Matter Expert. 
It is difficult to guess how many reviews he would get 
for publishing a list of objectives for English Literature 
1, Philosophy 4, Introductory Chemistry, or even 
Algebra, Geometry, and First Year Spelling. 

In summary, I have tried to examine the paradox 
between what everyone says to do — write behavioral 
objectives — and the environmental events that do not 
reinforce this behavior. I have suggested that a suitable 
environment might be created (1) by developing objec- 
tives which can easily be converted into program 
sequences and (2) by reporting program evaluation in 
terms of objectives that specify a stimulus and a re- 
sponse, criterion test items that measure achievement 
of objectives, and field test data that show actual 
achievement. These suggestions, it is hoped, should 
make writing objectives not only necessary and manda- 
tory, but also useful and rewarding. 



TASK ANALYSIS~^ANE OR BLESSING? 

Charles G. Latterner 

Charles Laherner discusses task analysis. Task analysis is the determination of what 
a person does, how he does it, why he does it and the skills involved in doing the 
work. Although it is a tedious, time consuming, expensive and difficult job, that is all 
too often sacrificed on the altars of the gods of false economy, task analysis is a 
necessary basis for preparing programmed instruction materials that can guarantee 
the return of many dollars in fully trained workers. 



When any group of individuals concerned with pro- 
grammed instruction get together to discuss the prob- 
lems, the techniques, the new ideas, or what not, the 
place of task analysis in programming generally comes 
up. As in all things concerned with programmed in- 
struction, there are varying opinions of the worth of 
conducting task analyses, and these opinions run from 
the decidedly positive to the decidedly negative, with all 
shades of opinion in between. There are those who feel 
that the effort involved in performing a task analysis 
is just too much for the benefits received. There are 
others who feel that a really good program cannot be 
constructed without a detailed task analysis. In other 
words, is task analysis a bane or a blessing? 

The concept of task analysis is certainly not new to 
most of us. Industrial Engineers have been studying 
jobs for years. Some training directors have used these 
task analyses in the development of training programs 
in fields that require workers to perform certain types of 
manual work. In many instances, however, training 
directors have not been too concerned about making 
full use of the data available to them, and so they have 
gone merrily on their way, implementing training pro- 
grams which usually do not meet the needs of the 
workers when they are put on the job. 

Just what is a task analysis? There are many an- 
swers to this question, and the definitions of task (or 
job) analysis run from the very simple to the very com- 
plex. Simply put, task analysis is the determination of 
what a person does, how he does it, why he does it, and 
the skills involved in doing the work. Actually, describ- 
ing task analysis in this way relates the subject to the 
whole demesne of training. It is, in fact, an important 
part of an evolving training technology, and there are 
those who feel that task analysis comes first in the 
chronological listing of a training technology. Some 
feel that task analysis is the base on which a training 
program should be built, regardless of whether that 
training program is of a conventional type or a pro- 
grammed instruction text. 

Why is it said that task analysis is the base on which 
a successful program can be built? Simply because 
unless we know what the person is required to do, how 
he is to do it, why he does it, and the skills involved in 



doing the job, we cannot hope to be successful in 
training him. From a task analysis we can easily estab- 
lish meaningful objectives; we can correctly delineate 
the threshold knowledge required in the student; we 
can accurately determine the behavioral changes we 
want to effect in the trainee; we can select proper sub- 
ject matter for programming; we can determine correct 
sequencing for teaching; we can specify appropriate 
training or programming techniques, and v/e can estab- 
lish real criteria for measurement. 

Before taking a closer look at how we can use task 
analysis in developing programmed instructional mate- 
rials, we should look a little more deeply into the whole 
process of performing a task analysis. 

We have established that a task analysis involves four 
elements: the what, the how, the why, and the skills 
involved. In finding out what a worker does, an analyst 
must consider two types of activity: physical responses 
and mental responses. Physical responses required of a 
worker in a job situation include such things as grind- 
mg or polishing materials; inserting, soldering, wiring; 
or performing a multitude of other physical activities. 
On the other hand, a worker may mentally plan, com- 
pute, judge, direct, or otherwise govern the expenditure 
of his own physical activity, or that of others, by a 
corresponding exercise of mental effort. 

In a job, a person may expend any combination of 
physical and mental effort required by the job to be 
performed. In finding out what the worker does, the 
analyst establishes the complete scope of the job by 
considering the total universe of responses demanded 
by the tasks to be performed. All of the physical and 
mental activities involved in the job as a whole, as 
well as in the individual steps that constitute the whole 
job, must be carefully considered. 

There is seldom, if ever, a job consisting of only one 
act, be it physical or mental. Most jobs consist of more 
than one act, or task, but even one task may involve 
different physical and mental activities on the part of 
the worker. It is up to the analyst to discover these 
tasks and report them in the most direct terms so as to 
develop a clear, concise, coherent, and complete de- 
scription of the job. 
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In order that the trained worker perform his job 
satisfactorily, it is essential that the various steps be 
taken in the appropriate sequence on the job. At this 
point, it is well to note that when training materials 
are prepared, the sequence of presentation may weU 
shift. It does not necessarily follow that the correct 
educational sequence is the same as the sequence fol- 
lowed on the job. However, at this point in the prepara- 
tion of a task analysis, the investigator is not necessarily 
interested in how the training course will be organized. 
He is only interested in reporting the job that the 
worker does so that when a program is prepared all the 
data will be available in clearly understandable terms. 

Having determined what the worker will be expected 
to do as the result of complying with the demands of 
the job, the investigator should next turn his attention 
to how the worker will respond. How the work is done 
concerns the methods used by the worker in accom- 
plishing the assigned tasks. As in the case of delineating 
what the worker does, the analyst should consider 
both the physical and mental responses required. 

Physically, the performance of tasks may involve 
the use of machinery or tools, measuring instruments or 
devices, other related equipment, the following of 
routines and procedures, and the movements of the 
worker himself. 

Mental responses to the demands of the job lie 
chiefly in the “know hov/” that must be applied to the 
tasks. This may involve the use of calculation, fonnu- 
las, the application of judgment, or the making of 
decisions. 

The investigator should assure himself that he has 
a complete g?^asp of how the job is done so that when 
the time comes to prepare the training program nothing 
will be overlooked or ignored. 

Of the four parts of our overly simplifled deflnition 
of task analysis, perhaps the most important is Why 
the task or job is performed. Why a worker performs 
his job is the purpose of the job itself and is indicative 
of flie relationship of the tasks that comprise the total 
job. In determining why the worker does what he 
does, the investigator outlines the scope of the job, or 
stated in another way, he determines the purpose of 
the job. It is here that the analyst must use the utmost 
care in ascertaining and recording the reason why each 
task is performed. 

The investigating analyst must do this for two rea- 
sons; flrst to clarify the overall purpose of the job; 
and second, to show the task relationships as the job 
progresses to the completion of the work cycle. Failure 
of the analyst to explain why an activity is undertaken 
will leave the impression that the job has not been 
fully reported, and this can result in ambiguity in pre- 
paring training materials. 



Finally, the analyst must consider the skills in* 
volved in the performance of a task or a job. Defining 
the skills involved consists of a listing and explanation 
of the basic factors that must be considered. These 
elements bring out the manual skills, knowledges, abili- 
ties, and other characteristics required on the part of a 
worker, whether that job is manual, craft, professional, 
clerical or any other type. The recitation of the skills 
required in successful job performance consists of all 
the information necessary to discriminate between jobs 
and to establish the degree of difliculty of any job or 
task. 

Generally, an analyst will concern himself with ac- 
curately reporting on certain elements, among which 
are responsibility, job knowledge, mental application, 
dexterity, and accuracy required. The skills required 
in a job have a direct and potent application in the 
preparation of a training program. 

Having had this brief over-view of task analysis, let 
us proceed to consider the question asked in the title: 
Task Analysis — ^Bane or Blessing? At tliis point, it 
would be well to consider the following excerpt quota- 
tion from “A Technology for Training Design” a Train- 
ing Evaluation Report published at Lackland Air Force 
Base in August 1962. The authors of the report were: 
Dr. M. A. Zaccaria, Mr. W. E. Driskill, and Col. W. D. 
Ceely, (1962): 

Lackland Military Training Center personnel who 

are directly concerned with training have become sensitive 
to the need for a different approach to training program de- 
vel'jpment. . . . this sensitivity has led to the development 
of a training technology based on five principles. 

The first requirement for effective traim'ng is a 

definition of job or after-training requirements; second, a trans- 
lation of these job specifications into specific measurable ob- 
jectives; third, the measurement of the extent that individuals 
possess the desired behavior and knowledge prior to train- 
ing; fourth, the development of training which is specifically 
relevant to achieving the training objectives and which em- 
ploys the most efficient methods available; and last, the ap- 
plication of a quality control program that insures that in- 
struction meets objectives and graduates effective performers. 

In determining the job requirements of the job for 

which a course is to train, a job or task analysis must be 
conducted. A job or task analysis should determine the be- 
haviors required in the every day job situation. . . . 

The job analysis should also include specific ex- 
amples of the kinds of activities for which the course is to 
train. These activities can serve as the basis for training 
exercises which assure that training is relevant to the tasks 
to be performed. 

A reading of the above excerpt calls attention to a 
significant fact, which is that the development of a job 
or task analysis is the first step in the development of a 
training program. From this initial effort flow all other 
training activities. 
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Perhaps we can answer our questions by asking 
some questions: 

How can we teach unless we know what we are sup- 
posed to teach? 

How can we describe the te rmin al behavior we want 
to effect in our students if we do not clearly know what 
they are supposed to do? 

How can we measure the effectiveness of our pro- 
grammers unless we develop a standard of judgment 
based on job requirements? 

Those who believe in task analysis as an important 
part of preparing programmed instruction materials 
contend that task analysis is a blessing to the pro- 
grammer, and that without performing a task analysis 
a really superior program can never be produced. 

Of course, in all honesty, it should be pointed out 
that there are negative aspects to using a task analysis 
as the basis for a program. First, it is tedious, time 
consuming, hard work. Much research must be per- 
formed for the more complex programs. Literature 
must be searched, workers must be interviewed and 
observed, and not just once, but over a period of time. 
Supervisors must be questioned and re-questioned on 
just what his workers are doing. (As a side note, there 
have been times when supemsors who really thought 
they knew what their subordinates were doing, really 
didn’t know until the facts of life were pointed out by 
the analyst!) Agreement must be reached among the 
workers, supervisors, analysts, and programmers as to 
what the job really requires and how to sequence the 
material. Sometimes the feelings of the ardent propo- 
nents of programmed instruction are deeply hurt when 
they are made to realize that perhaps certain things 
can be taught more effectively through other methods 
than by programmed instruction. 

It is generally recognized that the preparation of a 
programmed text is an expensive matter. It takes time 
and money to prepare a good program. Add to the 



initial expense the cost of preparing a complete task 
analj'sis and we may face the situation of not being 
able to afford the program at all. Business management 
generally looks on any training effort with a jaundiced 
eye. The comptrollers and cost accountaits are only 
too willing to sacrifice La Belle Training on the altars 
of the gods of false economy. Yet it should be recog- 
nized that the expendiutre of a few more dollars can 
guarantee the return of many dollars in assuring that 
workers are trained, fully trained, to do the job they 
are supposed to do. 

In summary we can conclude that: 

(1) the preparation of a task analysis is the logical 
beginning for preparing a programmed text. 

(2) the development of a task analysis greatly sim- 
plifies the definition of clearly stated, measurable, be- 
havioral objectives. 

(3) the work of program designers and program- 
mers is simplified by having a task analysis available. 

(4) by knowing exactly what a job requires of a 
worker, we can establish accurate estimates of desir- 
able threshold knowledge on the part of our student 
population. 

(5) we can estabhsh meaningful, measurable, stand- 
ard of performance on the job, after training. 

(6) program designers can more easily select the 
best programming technique to produce the desired re- 
sults in students. 

(7) changes in operational sequences or techmques 
can be quickly identified, and programmed instruction 
materials can be modified more easily. 

(8) there will be no doubt that the program will be 
designed to meet the exact needs. 

All in all, despite certain drawbacks, it seems rea- 
sonable to conclude that the development of a task 
analysis as a basis for preparing programmed instruc- 
tion materials is indeed a blessing for programmers. 



JOB ANALYSIS AND ITS APPLICATION TO TRAINING 

Joseph E. Morsh 

Joseph E. Morsh discusses the current Air Force method of job analysis, and the 
traditional uses of [ob analysis in preparing and revising job descriptions, in identify- 
ing training requirements, and in providing data for training standards and curricula. 
He also discusses some of the more sophisticated uses of job analysis which automatic 
data processing now makes possible. Finally he points out its indispensibility as the 
first step in the systematic preparation of programmed instructional materials. 



The value of job analysis as an adjunct to training 
has long been recognized in business, in industry and 
in government agencies. Job analysis is used to iden- 
tify and organize job content for writing or revising the 
job descriptions from which training requirements are 
inferred. Job analysis supports decisions concerning 
the extent to which on-the-job training may supple- 
ment or replace classroom training. Job analysis, by 
showing the relationship between jobs, may assist in 
reducing cross-training time. Job analysis provides in- 
formation for improving the coverage and designating 
the subject matter emphasis of proficiency tests used to 
evaluate the results of experience and training. Analy- 
sis of current jobs may be used to throw light on 
changes in training brought about by new equipment, 
new tools, and new methods of work. Job analysis 
provides essential data for the development of training 
standards or revisions of training curricula. Job analy- 
sis is especially important in identifying objectives for 
programmed instruction. 

All of these functions depend upon the availability of 
an effective method of job analysis. Before going on 
to discuss the applications, therefore, let us digress 
briefly to examine a method of job analysis that has 
been developed by the United States Air Force as the 
result of an intensive research program during the past 
four or five years. 

This method makes it possible to obtain work 
activity information quickly and economically — ^infor- 
mation which is dependable and current. The method 
makes use of an inventory which consists of a list of 
work activity statements or tasks which are categorized 
under major segments of the job, called duties. An 
attempt is made to select tasks which are about the 
same size, that is, at the same level of specificity. For 
practical administrative and analysis considerations, 
the size of a task is subjectively determined so that 
some 300 statements or less will cover a whole job. 
Each task statement is relatively independent of other 
task statements. A statement begins with an action 
word. It refers to a job activity, something the incum- 
bent does. Statements describing what the individual 
knows, understands, has responsibility for, is able to do, 
or is familiar with, are scrupulously avoided along with 
trite redundancies such as “when appropriate”, “as re- 



quired”, or “in accordance with prescribed procedures” 
and the like. 

When the incumbent completes the inventory, he 
checks the activities he performs. He also adds tasks 
which he does but which are not listed. In this way the 
inventory is kept current. When the instrument is 
revised, written-in task statements are included and un- 
checked tasks are deleted. Thus the inventory tends 
to present a true picture of what takes place on the 
job. In addition to checking performance or non-per- 
formance of a task, the incumbent rates all the tasks 
he does on one or more rating scales depending upon 
the purpose of the survey. He may indicate the time 
he spends on each task, task difficulty or importance, 
where the task was learned, on the job or in technical 
school, supervision required, or other information. 

Since the Air Force is scattered over much of the 
world, job inventories are administered by mail. In- 
ventory booklets for a related set of jobs are sent to 
testing officers who are located on practically all Air 
Force bases. These officers administer the inventories 
to groups of airmen incumbents or distribute in entory 
booklets to officers for self administration. The book- 
lets are then returned to a central office for analysis. 

Now let us discuss some of the applications of job 
analysis to training. As mentioned earlier, job analysis 
is used to discover job content for preparing job 
descriptions. From the job descriptions certain training 
requirements may be inferred. The validity of these 
inferences is directly related to the information avail- 
able in the job description. The training program may 
to a greater or lesser extent be dependent upon the ac- 
curacy and comprehensiveness of the descriptions of 
work performed. 

The end result of a survey using the task inventory 
method is a comprehensive and organized compilation 
of occupational information in quantified form. Vari- 
ous pertinent data are made available for the develop- 
ment of job descriptions. These data include such 
items as the proportion of the incumbents who perform 
each task and the average percentage of time spent on 
each task by those who perform it. Under present 
analysis procedures, the job descriptions for each 
skin level, apprentice, journeyman, supervisor, and 
superintendent, appear as lists of tasks arranged in 
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order of frequency of performance or proportion of 
time spent. As might be expected some tasks are 
unique to supervisors, others are most often performed 
by apprentices, while some tasks are equally often done 
by all skill levels. 

Job analysis may support decisions concerning on- 
the-job training. During a survey, incumbents may be 
asked to indicate whether or not each task was learned 
on the job or in the class room, or whether more or 
better training is needed for each task. Such informa- 
tion can be used to guide the training program. Those 
activities which are best learned on the job can be iden- 
tified and eliminated from the technical school training 
program. 

A systematic job analysis program, combined with 
electronic data processing techniques presently avail- 
able, will serve to identify the tasks which are common 
to two or more jobs, from which the relationship of the 
work performed in any one job to the work performed 
in any other job can be determined. Thus the amount 
and kind of extra training required in transferring to 
related jobs can be estimated. With the advent of auto- 
mation and advances made in electronic computation, 
the problems of cross-training workers in industry and 
in government agencies have become acute. When 
traditional jobs are eliminated or become reduced in 
importance, the former incumbents must be trained to 
do other work. The question is, what other work? One 
answer is that people can be most efficiently cross- 
trained in a minimum of time to the new job which is 
most similar to their old job. The Air Force method 
of job analysis makes it possible to determine which 
jobs are most ahke. Jobs which are being considered 
for cross-training can be surveyed and, by means of an 
iterative technique, jobs can be grouped in terms of 
similarity of content. In another approach, cross-train- 
ing time can be estimated and then jobs with minimum 
cross-training times can be identified. As automated 
personnel assignment procedures are adopted, the Air 
Force job analysis method by providing a great deal of 
quantifiable data will provide answers to certain of the 
problems of cross-training that are associated with 
these procedures. 

The effectiveness of job knowledge tests depends 
upon the completeness and accuracy of the job infor- 
mation available. Such tests must not only have good 
psychometric characteristics but must also have an 
appropriate pattern of content coverage. It is essential 
also, that the number of test questions designed to 
measure particular functions are directly proportional 
to the emphasis of those functions in the job. The 
development of properly weighted test outlines is there- 
fore a fundamental part of proficiency test construc- 
tion. The task inventory procedure, by obtaining such 



data as the number of individuals doing each task, the 
time required to perform, estimated complexity, and 
importance of tasks, provides a sound basis for the 
construction of test outlines, and since the basic data 
are readUy kept current, many of the problems encoun- 
tered in the construction of proficiency tests with 
appropriate coverage are thus eliminated. 

Job analysis is used to predict changes in training 
that will follow introduction of new equipment or 
methods. Here we must distinguish between two kinds 
of job analysis. The job analysis discussed thus far 
concerns the behavior of persons who are exerting 
physical or mental effort. The other kind of job analy- 
sis can be done in the absence of incumbents. The job 
analyzed is work that is required. A manufacturer of 
new equipment, for example, will specify in minute 
detail what must be done to operate or to maintain 
the equipment. Training, then, tends to be geared to 
the specified work requirements. Later it may be found 
that there is considerable discrepancy between the 
“work required” analysis and the job as it is actually 
done by an incumbent. It may be that some unneces- 
sary training has been given or that training has been 
inadequate for some aspects of the job. Since no job 
is probably completely new, information obtained from 
a careful analysis of current related jobs can be used 
to check and modify training where new equipment or 
procedures are involved. 

Job analysis provides information which is indis- 
pensable in developing training programs and in plan- 
ning and improving training course curricula. This 
function of job analysis has long been recognized as a 
tested principle of management. To train a worker 
for a job, the duties and tasks which the incumbent 
will be expected to perform must be discovered. On 
the basis of these, the skills and behaviors desired and 
the minimum essentials he must attain in training can 
be determined. A comprehensive program of job analy- 
sis will supply complete field information concerning 
the nature, scope, and requirements of all jobs for 
which the schools are training. A continuous flow of 
information collected in many places from persons ac- 
tually performing the jobs permits schools to keep their 
curricula vital and up-to-date. Schools are alerted to 
changes in the content and complexity of jobs and are 
enabled to keep abreast of technological innovations 
which induce changes in duties and tasks. The schools 
are made aware of variations in skills and knowledges 
required as well as alterations in the relative importance 
and time consumed by various aspects of jobs. Schools 
are thus able to keep courses realistic in terms of actual 
field job performance and requirements. 

Job information obtained from individuals who do 
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the jobs for which the school trains insures that school 
instruction in job performance will coincide with actual 
job performance. Graduates arriving on the scene of 
operations will find that they are properly qualified and 
that they need little or no additional on-the-job train- 
ing to become effective in their specialty. Thus the 
graduates and the supervisors on the job have confi- 
dence in the training program and in the technical 
schools responsible for it. The old attitude that there 
is “a school-taught way” to do a job and “a real way” 
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is eliminated. The school way guided by effective job 
analysis becomes the real way. 

The application of job analysis as the first step in a 
methodical approach to programmed instruction should 
now be obvious. For without job analysis there is no 
systematic way to determine what elements of subject 
matter or behavioral skills should be programmed. 
Finally, job analysis is essential in order to obtain 
criteria for revising and up-dating the entire instruc- 
tional program. 



DEVELOPMENT OF MEASUREMENT DEVICES FOR PROGRAMMED INSTRUCTION 

Robert M. Opdycke, Captain, U5AF 

Robert Opdycke discusses the use of tests before, during, and after the develop- 
ment of autoinstructional programs. He points out that proper use of tests will aid 
the programmer in identifying the skill level of the student prior to taking the program, 
help in the evaluation of the program through the development stage, and prove the 
program in comparison with other media of instruction. Problems encountered in 
testing progra.ms are also discussed. 



It appears that testing can play a more important 
role in programmed instruction than has heretofore 
been recognized. This paper suggests some testing 
procedures that could be used by programmers and 
program users before, during and after the development 
of an autoinstructional program. 

PRE-ENTRY TESTING 

A pre-entry test, as the name implies, is a question- 
naire or test ^ven prior to entry into a course of in- 
struction, or in this case, prior to taking a program. 
This test should be developed after a task analysis has 
been made and before writing the objectives. The task 
analysis will furnish the programmer with information 
on the final product (what must he know and what 
must he do). The pre-entry test should be designed to 
furnish information on the repertoire of a beginning 
student in comparison with the student who completes 
the instruction. 

As a result of a well constructed pre-entry test or 
questionnaire, the programmer should be furnished 
with feedback information which will enable him to 
identify the repertoire of the median of the entry target 
student population. Proper utilization of pre-entry test 
feedback data should aid the programmer by eliminat- 
ing much of the over-programming or imder-program- 
ming problems which all programmers find frustrating 
and difficult to prevent. 

The pre-entry test should be constructed with items 
ran^ng from simple to very difficult with a span that 
will enable the programmer to objectively identify the 
knowledge/skiL' level of the average student. This test 
inay prove to be a differentiating device; this is desired, 
because the students in the center of the curve of dis- 
tribution will furnish the programmer with feedback 
on where to set the level of the objectives for the 
program. 

This concept of pre-entry testing can be illustrated 
by a formula. “M” is the repertoire of an accomplished 
master or subject matter expert and “I” the initial 
repertoire of the entry student. The formula will then 
be: 

M - I = D 

where “D” is the difference between the novice and 
the master or “D” would be the behavior that must 



be changed or modified to reach mastery. Since no 
instruction or behavior change has taken place, then 
“D” becomes the training differential or training ob- 
jective. 

A task analysis will give a programmer the value of 
“M.” A pre-entry test will give the value of “I,” there- 
fore M — I — D, the training task or objectives of the 
program. With minor revisions the pre-entry test could 
serve as a pre-test for measuring the degree of learning 
and amount of behavior change when compared to a 
separate criterion post-test. 

Another useful benefit of the pre-entry test after the 
program is fully developed would be to employ the 
pre-entry test as a gate or branching device to place 
students at different stages in the program. A student 
who scores very low on certain questions in the pre- 
entry test should be given a remedial program or train- 
ing to bring him up to the level of the objectives of the 
program (up to the level of “I,” the average initial 
repertoire). Conversely, a student who excels on cer- 
tain items of the pre-entry test demonstrates prior 
knowledge and could bypass portions of the program. 

Many techniques are now in use to bypass, wash- 
back, or take students through a remedial loop in a 
program. These techniques are an internal part of the 
program, x’he pre-entry concept would be external; 
that is, removed from the program. 

The pre-entry test concept is largely hypothetical. 
Experimentation is needed on pre-testing techniques to 
develop a test that would take each student to his 
proper place in a program. 

DEVELOPMENTAL TESTING 

Developmental testing, debugging and revising are 
the keys to producing a good program which meets its 
objectives. Developmental testing of a program as 
visioned by the author violates most of the conventional 
concepts of measurement and evaluation. To list a few: 

1. A program teaches the criterion test. 

2. A program should skew a distribution curve as 
far right as possible on a criterion test, even to the ex- 
tent that students make perfect test papers. 

3. A developmental test should be so comprehen- 
sive and saturated that more time will be spent testing 
than teaching. 
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These three comments represent some of the con- 
cepts of developmental testing of a program. The 
philosophy of developmental testing is completely differ- 
ent than conventional testing. A developmental test 
does not test the student; it tests the program. A stu- 
den should not and cannot make errors, only the pro- 
gram can be blamed for errors. If the student fails, 
then the burden of responsibility rests on the program, 
never on the student. This philosophy is hard to accept, 
but is critically important; a developmental test must 
test the program and never the student. 

If the following outlines and procedures were used in 
preparing a developmental or criterion test, two valu- 
able tools should evolve: (1) a reduction in time to 
write a program, and (2) a sure knowledge of when a 
program has reached the criterion stipulated in the 
objectives. 

After the objectives have been written and before 
writing a single frame or lesson, a programmer should 
write and develop an extensive, comprehensive criterion 
test based on the criterion measurement contained in 
the objectives. Each and every objective and sub- 
objective should have one or more test items related to 
the terminal behavior set forth in the objectives. 

Here are some sample objectives and the related 
test items: 

1. Objective: The student will be able to define 
work, power, and energy. 

Test Item: Define work, power, and energy. 

2. Objective: The student will be able to weigh a 
given object on an analytical balance within .001 grams. 

Test Item: Weigh the sample object on the table 
at the front of the class and write down the weight. 

3. Objective: The student will be able to disassem- 
ble, clean, and reassemble the carburetor on a 1963 
Ford. 

Test Item: Using the tools provided, disassemble, 
clean, and reassemble the carburetor on the car. (Evalu- 
ation would be based on how well the car ran after the 
Ci..rburetor had been cleaned and replaced.) 

To reiterate, write the criterion test immediately 
after writing the objectives. The criterion test should 
test thoroughly each and every objective within the 
boundaries and terms stated in the objective. 

At this stage in program development the objectives 
are defined and a difiicult, saturated, comprehensive 
criterion test developed. Beforv=> starting on the pro- 
gram the programmer and training manager should 
determine the absolute grade, score, or level of per- 
formance on the criterion test that the training manager 
accepts as minimum passing or minimum for qualifying. 

The training manager has the prerogative of accept- 
ing the objectives and criterion test as written, adding, 
deleting or modifying to suit the training needs or re- 
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quirements as he sees them. Once this effort has a 
stamp of approval there is little that can be done until 
the program is completed. Programmers should make 
this point very clear to management. 

The objectives having been written, a criterion test 
developed and the training manager having accepted 
the efforts and set a level of achievement, the pro- 
grammer can now write the program. 

After developing a rough draft, the program should 
be given to one student and this student should be used 
to clean up semantical errors, grammar, punctuation, 
cueing, and the like. The individual student will serve 
the purpose of reducing frame error rate and very little 
else. After the first student has completed the program, 
he should take the criterion test; don’t grade or evaluate 
the test but set it aside. Give the program to another 
student and again perfect individual frames or se- 
quences. The second student should take the criterion 
test and these results also should be set aside. After 
approximately five students have completed the pro- 
gram, the frame error rate should be decreasing and 
frame sequencing should be fairly consistent. 

The programmer now has five criterion tests and a 
small sample of the student population. He should 
then score and analyze the criterion tests identifying 
the areas where the students missed criterion test items 
and the corresponding objectives. This small amount 
of feedback information should be used to rewrite and 
correct weak areas in the program. The assumption 
can be made that if small samples of students missed a 
certain area on the criterion test then the program did 
not teach this objective effectively. 

Next the program should be revised using the feed- 
back from the five individual students and a new draft 
prepared. Revisions and testing, using small samples 
of students, should be made untU the program is de- 
veloped to the degree that it is teaching to the level 
agreed upon by the programmer and training manager. 
When the program produces results with small student 
samples, 25 to 50 drafts of the program should be 
produced. With feedback results from 25 to 50 stu- 
dents, the programmer should be able to fill any small 
gaps that were missed by the smaller sample student 
groups. When a program is producing results with 25 
to 50 students at a time, the program should be fully 
developed and ready for publication as a first edition. 

Once the program has been fully developed, the 
criterion test can be disposed of and a shorter sampling 
test that is objective and easy to grade substituted to 
assure that students are completing the program and, 
if necessary, to identify students who are not perform- 
ing up to standard. 

To reiterate, the two-fold purpose of the develop- 
mental criterion test is to first enable the programmer 
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to develop a program by using feedback information 
from the test to fill any gaps that appear in the pro- 
gram and second, to recognize when the program is 
producing the criterion standards set in the objectives 
by the training manager. 

COMPARATIVE TESTING 

Comparative testing is the comparison of program- 
med instruction versus other mecfia of instruction. The 
end-product of comparative testing is statistical data 
that will prove one method superior to the other. Why 
compare? Psychologists have made comparison studies 
too numerous to mention; the evidence supports pro- 
gramming in a vast majority of these pilot experiments; 
furthermore, when a program undergoes development 
to produce a controlled and predictable behavior 
change that is tried and validated by statistics on the 
criterion test, then why is a comparison study neces- 
sary? Unfortunately, programmers will face some man- 
agers and supervisors who regard programming as a fad 
and who insist on comparative statistics. 

Comparative testing and comparative statistics are 
so flexible that concrete procedures cannot be set for 
any given situation. Here are a few guidelines that can 
be followed. 

The criterion test that is used during the develop- 
mental tesdng of the program can be used as the com- 



parative te.st if the objectives of conventional instruction 
are the same as the objectives of the program^ If there 
are differences between the two, then a separate test 
should be developed containing items that pertain to 
both methods of instruction. A programmer should be 
equipped to furnish answers and data to management 
along the foUov/ing lines: 

1. Method and supporting data to assure that there 
is no significant difference between student groups be- 
fore instruction. 

2. Comparison of test scores. 

3. Retention test comparison. 

4. Time comparison. 

5. Student preference. 

6. Findings and recommendations. 

If a good criterion test is used to develop the pro- 
gram, there is no reason why programmed instruction 
should not prove superior in all of the above points. 
Programmed instruction is the only educational tech- 
nology that can safely say that it is developed to com- 
pletely control the learning situation and is geared to 
each individual student. At some point in the not too 
distant future, programmed instruction should also be 
able to guarantee every student an “A” and also furnish 
knowledge and skills that are far above our present 
educational methods. 



QUALITY CONTROL PROBLEMS AND THE SELECTION OF WELL-CONSTRUCTED PROGRAMS 

Shirley B. Bifterlich 

A step-by-step procedure for developing programs of high quality, from detailed 
preliminary planning to testing and revising, is presented by Shirley Bitterllch. Practical 
suggestions for judging programs by the prospective user are Included. 



The goal of every programming company is to pro- 
duce a program of high quality. K the effort is success- 
ful, the result is a deceptively simple piece of work: 
simple, because it flows smoothly and teaches easily 
and well; deceptive, because it reflects in no way, except 
in its apparent simplicity, the problems which have been 
faced during its construction. This paper will discuss 
some of the problems which arise during the precon- 
struction and construction phases of a program, and 
also some of the criteria which may be used to judge 
the degree of excellence of a program. 

Detailed preliminary planning is essential. The sub- 
ject-matter area must be thoroughly researched. In the 
writing of an academic program, curriculum surveys 
are made, textbooks are reviewed, current research in 
the topic area is studied, and a subject-matter specialist 
consulted during all stages of planning and production. 
Only after the subject area has been thoroughly ex- 
plored can the topic outline be drawn up, analyzed, and 
broken down into the specific objectives. 

Preliminary planning serves a three-fold purpose: 
to validate the subject-matter content, to ensure that 
all necessary behavioral objectives are taught, and to 
guide the construction of a properly sequenced pro- 
gram. 

The preconstruction phase entails careful considera- 
tion of many diverse factors which have direct bearing 
upon quality control. Research in the subject area can 
guarantee the technical accuracy of the broad content 
of a program, but the selection of the appropriate inter- 
pretation requires a detailed analysis of the material. 
In order to present the subject matter in a manner 
which is suitable to the intended grade level, it is often 
necessary to present highly complex concepts in simpli- 
fied form. The simplified version must still be consistent 
with the accepted concept or theory, and there is always 
the inherent danger of oversimplification. If it were 
easily possible to state a comple theory in simple terms, 
the theory would not be complicated in the &st place; 
yet, this is a problem the prognmmer must solve. 

The quality of a program is directly dependent upon 
the reliability of the subject matter. The overall de- 
velopment of a highly involved concept must be simple 
and logical, but, at the same time, each frame must be 
able to stand alone as presenting accurate information 
under tight stimulus control. 

Experience has shown that the production of a pro- 
gram of high quality requires the diverse talents of 



many skilled individuals: the psychologist, as the au- 
thority on programming techniques; the subject-matter 
consultant, as the authority on content accuracy; the 
programmer and editor, as the coordinators and con- 
structors of the program. 

The subject matter of any program must be pre- 
sented in a writing style which is appropriate for the 
intended grade level. The extent of the student’s vo- 
cabulary must not be overreached; yet the level of word 
content must be high enough so that the student does 
not feel that the program is in any way patronizing 
his intelligence. Any program, on any subject, written 
for any grade level, should be grammatically impec- 
cable. 

A program requires iutense concentration on the part 
of the student. Although attention span increases with 
increasing maturity, any program written for any level 
must change pace in some fashion to provide a break 
in the concentration required by the program, but no 
break in the continuity. The use of humor is reason- 
ably appropriate in programs written for lower grade 
levels, yet the humor should be related to the content 
of the program. Programs written for higher grade 
levels may more appropriately use other means of pro- 
viding a change of pace. This may be done by mo- 
mentarily lowering the level of response called for in 
some frames. A sequence which develops a diflScult 
concept and requires maximum concentration may be 
followed by a few such frames; then the tempo and 
level of response can be iucreased as a new sequence 
is developed. 

The programmer must concentrate equally on the 
subject matter and the programming techniques by 
which the subject matter is presented. The frame should 
include no irrelevant material; every bit of information 
given should be essential for the student to be able to 
make the response. If the student does not need to use 
any of the information, it should not be in the frame. 
Conversely, the response should be a direct check of 
information given in the stimulus. The response called 
for in the frame should always be the critical response. 
If, for example, the stimulus states that X=Y, and the 
equality of X and Y is the critical piece of information, 
then the student should be called upon to respond in 
some way concerning the equality of the two rather 

than to complete a statement such as X = If the 

emphasis is on the fact that Y, a new fact or concept, 
equals a previously taught or known X, then the 
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response should be in tenns of Y rather than equality. 
The selection of the critical response is an extremely 
significant and extremely difiScult part of the art of 
programming. A programmer frequently writes the 
response first, then tailors the stimulus to fit the 
response. 

The relationship between response and stimulus is 
reversible; that is, the response called for should be 
the one of primary importance, and the stimulus should 
present the information so that the response is under 
stimulus control. If any response other than the one 
given in the confirmation can rightfully be made, the 
response is not under stimulus control and the stimulus 
is improperly stated. The stimulus portion of the frame 
should be constructed so that one, and only one, re- 
sponse is appropriate. 

If the programmer is satisfied that the response 
called for is the critical one and is the only response 
which is appropriate, he must still examine the stim- 
ulus to see if there is any way the student can make 
the right response for the wrong reason. In the initial 
development of a new concept, it is frequently neces- 
sary to cue the response rather heavily, so that the 
frame is in essence a copy frame. As a teaching se- 
quence develops, the intentional prompts are gradually 
withdrawn. However, the programmer must constantly 
be on the alert for internal cuing which allows the stu- 
dent to respond correctiy because of some extraneous 
stimulus. An unintentional internal cue may take 
many forms. It may be a grammatical cue, such as 
placing “an” or “a” before a blank. If one of two al- 
ternative responses starts with a vowel sound, the stu- 
dent can automatically make it if he sees “an”. If be 
sees “a”, he can, again automatically, respond with 
the word which starts with a consonant sound. The 
same grammatical construction may always have been 
used to state a certain fact. In this case, the student 
may be responding to the phraseolo^ of the stimulus 
rather than to the information given in the stimulus. 
Many programs necessitate the use of diagrams, and 
this compounds tlie problem of internal cuing. Is the 
student responding to the intended stimulus, or is he 
selecting his response on the basis of an arrow that 
points to the right rather than to the left? Diagrams 
should be varied, should contain as few cues as possi- 
ble, and should be used only when it seems impossible 
to present the stimulus adequately without one. 

A program must be thoroughly tested before it is re- 
leased. A preliminary pilot testing of the first section 
of a program is frequently made to check on the cor- 
rectness of the writing level. It is essential, especially 
when writing for lower grade levels, to test such factors 
as depth of vocabulary and difficulty of grammatical 



construction as well as clarity of presentation. A thor- 
ough analysis of the testing results obtained on the 
complete program must be made and the program re- 
vised accordingly if necessary. 

What criteria may be used to aid in the selection of 
a well-constructed program? Ideally, the entire pro- 
gram should be read from start to finish. The prospec- 
tive purchaser can then best judge whether the pro- 
gram covers the intended topic area and in fact teaches 
the essential material in the proper sequence. A study 
of only the first part of a program usually does not 
give a true picture of the program as a whole. A pro- 
gram presents material in a way which is new to the 
student. He must first learn to use this new tool; con- 
sequently, a program frequently starts slowly and may 
seem to present its material in an unnecessarily ele- 
mentary fashion. Once the student has been given 
time to accustom himself to these new techniques, the 
tempo can, and does, increase. If individual frames 
are inspected, each one can be analyzed as to its in- 
ternal construction: relevancy of information, appro- 
priateness of response, stimulus control, internal 
cuing, etc. A block of several frames may be studied, 
to look for the orderly development of ideas. Does 
each response demonstrate an increase in the student’s 
knowledge? Is the concept developed fuUy? Is the stu- 
dent given numerous examples which ffiustrate the 
application of the concept or rule? Is he taught to use 
his new knowledge? 

Factual evidence should be presented which demon- 
strates that learning occurs as a result of the use of the 
program. For example, a comparison of the scores 
obtained on identical subject-matter tests, adminis- 
tered prior to and following the use of the program, 
should show a significant gain in the student’s knowl- 
edge. 

The program should also be viewed from the stand- 
point of its interest to the student, and a careful study 
should be made concerning the manner in which the 
subject matter is presented. The program should include 
a variety of frame styles so that learning without bore- 
dom is the result. However, the program should be 
well balanced: enough variety to provide interest, but 
not so much that the student can become confused from 
trying to determine, not what the response is, but how 
he is supposed to make it. 

Above aU, one should look for the deceptive simplic- 
ity of the program. Some of the features wffiich may be 
used as criteria have been mentioned; if it progresses 
logically, has good stimulus control, teaches easily 
and thoroughly, covers its intended topic area, and 
holds the student’s interest, it is a well-constructed 
program. 
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SKINNER TECHNIQUE IN REINFORCED LEARNING AS APPLIED TO REMEDIAL SPELLING 

Glenn H. Rogers 

Glenn H. Rogers reports an experiment to improve the spelling ability of college 
students. An extensive readiness program was followed by the use of programmed 
material. A small sample of five students improved spelling ability from a mean of 
34% on the pre-test to 99% on the post-test. 



This experiment was conceived as an attempt to im- 
prove grade averages of a group of students in basic 
English classes who were substandard in spelling. 
Subjects were selected from volunteers recommended 
by instructors of these classes. The final group of sub- 
jects consisted of those whose spelling responses placed 
them below the 50th percentile as measured by Fresh- 
man’s spelling tests. 

The technique known as “reinforced learning” devel- 
oped by Dr. B. F. Skinner of Harvard University was 
utilized in this experiment, after both visual and audi- 
tory conditioning of the subjects. The Skinner linear 
approach was applied through the use of auditory stim- 
uli presented via tape recorder. 

In a series of pre-tests it was discovered that students 
with spelling ability below the 50th percentile had nearly 
100 percent \dsual imperceptibility, and in most cases, 
faulty phonetic perceptibility. Subjects could neither 
see mentally the simplest symbolization, nor could they 
accurately identify auditory stimuli with the proper 
symbolic counterparts. This was especially true where 
words contained vowels or vowel blends. This phon- 
etic anolamie is present in areas where the production 
of English vowel sounds, in some cases, is distorted. 
Visual imperceptibility was established as a basic fault 
when subjects, while blindfolded, were unable to ac- 
curately draw simple figures such as a cross, square, 
circle, or triangle on a chalkboard. 

Having determined the presence of both the visual 
and auditory limitations in the subjects it was decided 
to incorporate a readiness program consisting of phon- 
etic and visual training before applying the Skinner tech- 
nique. Visual imagery was sharpened by having sub- 
jects “see clearly” familiar objects about their homes, 
rooms, garages, and yards. Each was asked to shut his 
eyes and mentally “take a walk” around his home 
grounds, to enter the house, to go carefully from room 
to room, seeing each familiar object m detail. This 
“tour” was accompanied by audible descriptions, with 
frequent questions by the tester designed to elicit detail. 

Vowels and consonant symbols were then placed on a 
blackboard and subjects were asked to reproduce each 
several times, visualizing the symbol mentally, and 
with clarity. Individual symbols were then combined 
first into short words, then into longer ones. The 
practice of visualizing clearly was continued throughout 



the experiment, and preceded each group of presenta- 
tions during the programming. 

Phonetic identifications are important in a spelling 
lesson which is presented as auditory stimuli. Whether 
the stimuli originates as “imagery” or as auditory vi- 
brations impinged upon the tympanic membrane is 
not important. In either case accuracy in spelling de- 
mands that the auditory phoneme or combination of 
phonemes be properly identified with the symbolic 
counterparts in all the variegated forms found in the 
English laiiguage. 

Example: the Phoneme “u” is the same in “stew”, 
“too”, “soup” and “through”. 

The knowledge that English is a polyglot language 
and thus contains many words that do not even ap- 
proximate the sounds being represented does not as- 
suage the harassed student to whom “boat”, “mote”, 
“stow”, and “though” all look pretty much alike vowel- 
wise. 

As a part of the readiness program in this experi- 
ment these students were given a basic lesson in phon- 
etics, with repeated drills on the vowel phonemes. 
These phonemes were presented audibily at the same 
time their symbolic counterparts were presented vis- 
ually, in all existing variations. Once the students 
learned that there are seventeen basic vowel sounds, 
not five as they had supposed and three basic diph~ 
thongs, the establishment of sound identification to 
vowels and vowel combinations was not a difiicult one. 
Addition of consonants followed quickly. 

Students were considered “visually ready” when 
they could reproduce clear mental images. They were 
considered “auditorily ready” when they could recog- 
nize, identify and reproduce vowels, consonants and 
unphonetic combinations with accuracy. The entire 
readiness training consumed only four periods. 

A readiness program for a remedial spelling project 
cannot neglect the semantic component. In the list of 
words selected for presentation, most were found to be 
in colloquial usage, and needed only contextual asso- 
ciation to explain their meaning. This was provided 
during the initial pronunciation, and further presenta- 
tion of the words. 

Presentation of the Constructed Response 

Five students were used in the experiment. A list of 
1000 hard-to-spell words compiled from Shaw (1961) 
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was used. Initial tests were given and scored. Fol- 
lowing phonemic and visual readiness training, the 
linear method of constructed response‘s was used to 
present 200 words classified as “difficult to spell”. 

These were presented via a mechanical device 
called a “Cyclo-teacher” published by Field Enter- 
prises Educational Corp., publisher of World Book En- 
cyclopedia. This consists of a revolving disk on which is 
placed a work wheel containing fifteen constructions on 
each side. The Subject reads the construction, writes 
his answer in an appropriate box, turns the wheel and 
checks with the correct word which appears in a win- 
dow. Each word is presented three times in variable 
patterns designed to emphasize the “difficult” segment 
of the word. 

Example: 

Construction Response 

His N ghbor made periodic visits to check his wel- 
fare. ne/ghbor 

His closest n ghbor lived three miles away. 

ne/ghbor 

He had been a good neigh — for twenty years. 

neighft(?r 

Subjects were required to repeat each wheel in which 
an error occurred. 

The major segment of this training consisted in the 
presentation of words via auditory stimuli, using a tape 
recorder. The linear approach was used, and emphasis 
on repetition, frequency, and reinforcement of im- 
mediate reward was applied. 

Words were grouped germanely, when possible, into 
both phonetic and unphonetic patterns. Words in a 
group were related pictorially. This type of group- 
ing, following the principles of repetition and frequency, 
aided in establishing the visual responses. 

Example of pictorial grouping: 

ei words ence endings tion endings ur endings 

ie words ance endings cion endings or endings 

ents endings sion endings er endings 

ir endings 

Both General American and phonetic pronunciations 
were used. The phonetic pronunciation, it was felt, 
aided in establishing auditory identifications with the 
correct visual image. Example, the word “analyze” 
would be given normally and also “anal-wize” to em- 
phasize the “y” spelling. The word “accommodate” 
would be given “ac’com’mo’date” to emphasize the 
phonetic breakdown. Semantic components were not 
ignored as identification factors. Where necessary, a 



word was used in an appropriate simple sentence to 
establish an interpretation and meaning. 

In the presentation of the words, frequency was con- 
sidered an important factor. In a given word the fre- 
quency was accelerated at the beginning of a group 
presentation, for example: 

AABABBCCBABCDD 

Each word was presented four to six times in a group- 
ing. 

Students worked both individually and in groups. 
The taped presentation can be considered auto-instruc- 
tion as it was not necessary for an instructor to be 
present. Each word group was given a number, which 
was easily located on the counter-dial built into the tape 
recorder. A word group was repeated by a student only 
when he felt this necessary. Following the completion 
of each word group, a final test of the word list was 
given. 

Results 

After an average of 30 hours auto-instruction, which 
followed pre-conditioning, the five subjects showed an 
average net gain of 64.5 in correct responses. Students 
worked individually and in groups, and were able to 
complete the 88 groups in eight weeks, operating the 
tape recorder one hour per day three to four days a 
week. Because of the limited number of tape record- 
ers available, students worked at night, on week-ends 
and other odd times. 

Subjects moved from initial average of 34.36 correct 
responses to 98.72 correct responses in the experiment. 
The greatest gain was from 27.7 to 99.3. Subjects moved 
through the 88 groups of 1000 words without an inter- 
mediate testing period. The final dictation was given 
following the completion of the eight tapes used in the 
experiment. 

Retention 

The weakness of this experiment was in the inability 
to make an accurate evaluation of retention. At the 
cios“ of the experiment the Spring semester was over, 
and ochool was out. It is to be noted, however, that 

Table I 

Reduction in error during auto-instruction 



Student 


% Error 
Initial Test 


% Error 
Final Test 


A 


53.8 


0.009 


B 


61.2 


1.4 


C 


72.3 


0.007 


D 


68.7 


1.6 


E 


122 


1.8 
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no intermediate testing was applied, and in the final 
test, the words presented at the be^nning of the experi- 
ment showed no greater tendency toward inaccuracy 
than words presented nearer the close of the experiment. 

Negative Correlation With Mental Ability 
Table III shows correlation of “mental ability” with 
percent of initial error. Otis Mental Ability tests: 



Table II 

Percentage of gain in correct responses during 
auto-instruction 



Student 


% Correct % Correct 
Fined 

Initial Test Test 


Net 

Gain 


Hours spent 
Auto- 
instruction 


A 


46.2 


99.1 


53.7 


30 


B 


38.8 


98.6 


59.8 


32 


C 


27.7 


99.3 


71.6 


25 


D 


31.3 


98.4 


67.1 


35 


E 


27.8 


98.2 


70.4 


32 



Glenn H. Rogers 

Gamma AM were presented to these subjects. Results 
of this testing do not indicate any significant positive 
correlation between low spelling deficiency and sub- 
jects’ ability to make correct responses on the Ods 
MA tests. It is to be noted that all IQ scores, as indi- 
cated by the Otis tests, fall within the range established 
as “normal” i.e. between 90 and 110. 

Table III 

Correlation between initial test results and scores of 





Otis M/A tests. Gamma AM. 


Student 


% Correct 


% Correct 


Raw Score/ 




Initial Test 


Final Test 


I.Q* 


A 


46.2 


99.1 


51/109 


B 


38.8 


98.6 


43/101 


C 


27.7 


99.3 


42/100 


D 


31.3 


98.4 


44/103 


E 


27.8 


98.2 


42/100 
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ADJUNCT PROGRAMMING 



M. A. Zaccflria and Lt. Charles F. Adams, USAF 

M. A. Zaccaria and Charles Adams start by questioning the importance of written 
responses and immediate knowledge of results. They state that these features of 
programmed instruction are not always necessary for effective learning. On the basis 
of Air Force research, they raise considerations of economical, efficient adjunct pro- 
grams. An adjunct is basically a series of objectives in question form. The questions 
are given to the student and direct his attention to the key points in the content of the 
basic narrative text. 



In spite of the fact that most experts in the field of 
programmed instruction seem to explicitly state or 
imply that real learning takes place only with a private 
tutor or through programmed instruction, hundreds 
of millions of people have been learning knowledges 
and skills in the past without either method. These 
millions of people have been learning from books, 
classroom lectures, by trial and error, and such other 
methods as all experts in programmed instruction abhor. 
True believers in programmed learning must come to 
grips with the hard reality that the vast majority of 
learning in the past, and for a long time in the future, 
will take place without benefit of programmed instruc- 
tion. 

How can we improve the efficiency of education and 
training that will continue to be accomplished with un- 
programmed instructional materials? How can we in- 
crease the amount of learning that takes place without 
the assistance of or through the media of programmed 
instruction? How can we increase the awareness and 
appreciation of the principles of programmed learn- 
ing with students, parents and administrators? And 
how can the vast majority of education using unpro- 
grammed materials be improved until it is consigned 
to oblivion through complete replacement by program- 
med instruction from kindergarten to the Ph.D? 

Let us begin by questioning some of the hallowed 
principles of programmed instruction. Linear program 
theory is based on several assumptions that have not 
always been substantiated by research or evaluative 
studies. For example, studies conducted at Lackland 
AFB, University of California, at Los Angeles, and 
Stanford University indicate the students do not have 
to respond to each frame for effective learning; in 
fact, no written response at all may be as effective as 
making constructed answers. This finding has both 
theoretical and economic implications. 

The assumption that a stuocnt should be reinforced 
by being shown the correct answer each time after he 
responds has been questioned by recent studies. The 
application of this finding can make for more efficient 
use of time and equipment. 

Some amazing findings have evolved from informal 



developmental studies. Programs have been construc- 
ted that merely divide the written material into small 
segments with a key word to be constructed by the 
student. Even though the material is not considered 
to be well sequenced or programmed, the program 
teaches quite effectively. In fact, one may learn to 
make appropriate responses in a criterion test, even 
when he practices the irrelevant responses for which the 
programmer has left a blank. It appears that the mere 
breaking down of information calls for enough attention 
of the learner so that he learns, at least in some kinds 
of situations. 

Recent studies at Lackland Air Force Base have in- 
dicated that an adjunct program can be less costly to 
develop and administer and yet produces as much learn- 
ing as a conventional program. This program asked 
questions of the student for each paragraph of text. The 
learner was forced to analyze the printed material and 
to write his answer to a question that tested his knowl- 
edge of the paragraph. Student completion time was 
half that of the corresponding linear program. The 
questions, in effect, became the course objectives. We 
hypothesize mat the student realized that tiie questions 
were his objectives and motivated himself to learn them. 

There are many possible ways of developing adjunct 
programs to already existing texts. The main service 
these adjuncts would provide is to call attention to the 
educational or training objectives. One method of pre- 
senting an adjunct is to list all of the relevant test 
questions for a text. A refinement of this is to indi- 
cate on which pages and in which paragraphs o£ printed 
materials the answers can be found. A further refine- 
ment could be to develop branching or by-pass ques- 
tions. Another technique is to develop a linear pro- 
gram to give students certain basic knowledges such 
as terminology; this could be used in conjunction with 
questions on the text. Although the authors prefer to 
use constructed response questions, they know of no 
reason why appropriately developed multiple-choice 
questions could not be developed. 

There are a number of advantages for using adjunct 
programs over the conventional linear, intrinsic, and 
mathetical types. Adjunct programs are less costly to 



develop. As content changes, they can be readily mod- 
ified or revised and be easily and quickly published. 
They are more relevant to everyday work and learning 
situations and there is less fear of providing students 
with a crutch. One cannot ignore the fact that there 
would be less resistance to this technique on the part of 
both teachers and students. 

The authors realize that adjuncts are not necessarily 
new ideas. They are not sure that their concepts are 



^A. A. Zaccaria and If. Charles F. Adams, USAF 

drastically different from those of Pressey’s back in 
the early 1930’s or of the workbook. They are not as 
concerned, however, with presenting new ideas as they 
are with making education and training more effective. 
This is a pragmatic approach. Anything that works is 
good. Adjuncts are not panaceas for all instructional 
problems. They are merely presented as one possible 
practical and economic consideration for effective 
learning. 



A CONSIDERATION OF SOME OBJECTIVE PROPERTIES OF LINEAR PROGRAMS 

B. W. Yaeger 

Bernard Yaeger points out that programmers have few objective tools available 
during the initial preparation of material, depending instead on student tryout infor- 
mation to shape the eventful field version of the program. He discusses several experi- 
mental procedures and their apparent eflFectiveness. 



One of the problems facing a program writer as he 
prepares a linear type program is the lack of objective 
measures to use as yardsticks or guide lines. Although 
an experienced programmer does develop some “intui- 
tive feel” for his task, this feeling should be rendered 
objective and preferably quantifiable. The program- 
mer begins by shooting in the dark, particularly in 
regard to step size. Normally this is a very subjec- 
tive process and is based upon the programmer’s sub- 
jective opinion of a “step” and of a student’s ability 
to take that “step”. Before these opinions can be 
verified by tryout on actual students, the programmer 
must proceed by his intuitive feelings, in regard to step 
size, rate of progress, repetition, relevancy of responses, 
etc. The approach that an appropriate size of step is 
that size which yields a 5% error rate is useless at 
this point. What the programmer needs is before-the- 
fact information on step size rather than after-the-fact 
results obtained during the tryout. Besides an ob- 
jective measure of step size, the programmer also 
needs to know the amount of repetition or response re- 
dundancy to build in and ideally some measure of 
diffi culty, if one can be found. These objective mea- 
sures v/ould also be helpful in preparing corrective 
sequences or in preparing a series of tracks for multi- 
track programs. 

With multi-track programs we can accommodate 
students of different abilities, backgrounds and experi- 
ence levels more effectively. For example, we could 
provide a fast track program at a high level, one al- 
ternative sequence of an intermediate level and a third 
alternative sequence at a low level. We could use then 
the same program for a wider range of students and 
in a wider range of situations. Also, material could 
be used for a variety of different purposes, such as 
original learning, review, and for refresher training. 
We could write this kind of program with much 
greater accuracy by using the objective measures as a 
guide; that is, by using objective measures, we could 
improve the appropriate parameter with assurance that 
other parameters were not at fault. This would assist 
the programmer by permitting more efficient use of his 
time and by providing him some assurance that the 
program was near the target. Effective measures could 
be used before student tryout, and would be very help- 
ful when students are not readily available. 



•ERIC 



One of our objectives in this type of research is our 
interest in the utilization of programmed material in 
teaching machines. The closer we are on the first try, 
the fewer times we will have to revise material for ma- 
chine presentation. This process will almost always be 
more expensive than the rough typing on index cards 
approach. Currently, the advice to the programmer 
continues to be, “write some frames and try them out” 
and let the student dictate answers to these problems. 
But this approach is likely to require an impractical 
number of trials. So what objective measures can we 
use before-hand? Let me say first that any before-the- 
fact measures will not be completely sufficient and 
usable by themselves. Every program need to be 
tried and verified on actual students. This tryout gen- 
erates an error rate measure of step size and a final 
test indicates whether or not the student has learned. 
Better still is on-the-job success. But obviously we can- 
not wait for the on-the-job performance to verify a 
program or to indicate that our seat-of-the-pants feel 
for step size, repetition and difficulty was correct. What 
is needed is one or more objective measures, inde- 
pendent of student performance, which can serve as 
guide lines in program preparation.. 

We at Honeywell have begun some exploratory re- 
search in this area. This research is far from com- 
plete, but thus far has yielded some interesting data. 
Some of our initial ideas we are now fairly certain will 
not work, and we have some information on other ideas 
that look like they will work. The first of these is pro- 
gram density. This is an approach proposed by Edward 
L, Green of Dartmouth in his book “The Learn- 
ing Process and Programmed Instruction” (1962). 
Program density is the ratio of the number of differ- 
ent responses to the total number of responses made. 
For example, if each response term was called for 
twice, the density ratio would be .50. If no response 
was called for more than once, the ratio would be 1.0, 
or unity. Green proposes this measure as an indica- 
tion of the rate at which new material is introduced. 
We have interpreted it as a measure of repetition, sud 
as an indication of difficulty. Green found that program 
density was significantly related to error rate (r=0.47, 
P.005). The denser the program the higher the error 
rate. 
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The second independent measure we looked at, and 
this is one which does not appear worthwhile, is the 
number of responses per frame. We looked at this as 
a measure of step size to test the hypothesis that a 
step is the distance between responses. The third mea- 
sure again relates to step size and is a count of the 
number of sentences per frame or per step. Is step size 
measurable in terms of number of words? The fourth 
measure is an indirect measure of difficulty. We asked, 
“Is it possible that readability is related to difficulty, 
error rate, etc?” We selected the Gunning (1952) read- 
ability index or “fogcount” as the objective measur- 
ing device. This is an empirically derived formula 
which counts the average number of words per sen- 
tence and the number of 3 syllable words per 100, 
and yields an index which is related to school grade 
equivalent. 

Let’s examine several samples of published pro- 
grams with these independent measures. 



Example 1: 
Tide- 


“Plane Geometry” — 106 


Author — 


frame sample 
Edward B. Curds 


Published by — 


Encyclopedia Britannica 


Population — 


Films TEMAC 
9th grade 


Density — 


.62 (all responses; numer- 


Responses/frame 


ical and verbal) 

.51 (verbal response only) 
1.13 


Sentences/frame 


1.71 


Readability 


8.33 



Note the density ratio of .62 for all responses and 
.51 for verbal or word type responses only. This shows 
an average of 2 responses for each response term used. 
Note also the readability index of 8.33. The author 
aimed for a 9 th grade population and his readability 
index indicates that he hit the mark, i.e., between 8th 
and 9th grade. 



Example 2: 



Tide- 

Author — 
Published by — 
Population — 

Density — 
Responses/frame 
Sentences/frame 
Readability 



“Descriptive Statistics” — ^78 
frame sample 
Teaching Machines Inc. 
TMI— GroUer 

College population and bright 
high school students 
.38 
2.16 
2.55 
12.0 



B. W. Yaeger 

The TMI program shows a lower density, .38, but 
note the number of responses and number of sentences 
per frame. They are both higher than the TEMAC 
program. The readability index of 12 or 12th grade 
indicates that the TMI authors also hit their mark as 
far as reading difficulty is concerned. 



Example 3: 



Tide- 
Author — 



“The Analysis of Behavior” — 
Set 1-3 inclusive (114 frames) 
Holland, J. B. and Skinner, B. 



F. 

Published by — McGraw Hill 

Population — College 

Density — .33 

Responses/frames 1 .29 

Sentences/frames 1.25 

Readability 14.9 

“The Analysis of Behavior” — Set 1 only (54 

frames) 

Density .27 

Responses/frames 1 .40 

Sentences/frames 1.25 

Readability 1.25 



This is probably one of the best-known programs in- 
cluded in our analysis. Note the still lower density 
ratio. Also note the sentences per frame rate. When 
Holland and Skinner proposed the use of small steps, 
they meant small! Out of 100 frames only 25 have 
more than one sentence each. Contrast this with the 
TMI program which averages 2.55 sentences per frame. 

Let’s look at a few others also. 



Example 4: 



Tide- 
Author — 
Published by — 



Population — 
Density — 
Responses/frame 
Sentences/frame 
Readability 



“Earth” — 36 frame sample 
AIM Staff 

Accelerated Instruction Meth- 
ods 

179 N. Michigan Ave. 

Chicago 1, Illinois 
Jr. High School 
.39 
1.28 
2.11 
7.17 



Note density at .39 level and readability at the 7th 
grade level. Here again the authors hit their population 
target as measured by the readability index at any 
rate. 
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Example 5: 



Title- 

Author — 
Published by — 
Population — 
Density 

Responses/frame 

Sentences/frame 

Readability 



“AC and W Radar Repair- 
man ” — 97 frame sample 
Inman & Kelley 
ATC— Keesler AFB 
ATC Radar School Students 
.23 
1.57 
1.5 
10.4 



Final Tryout — 20 students 
Error rate average — 2.5% 

Final test average — ^91.56% correct 



This is a course developed by ATC in-house instruc- 
tor personnel. Note the low density ratio compared to 
the first three programs but note also the low error 
rate. The student population were all high school 
graduates so the readability figure of 10th to 11th grade 
could be handled easily. 



Figure 1 Summary of Program Analysis 

Holland 



TEMAC 


TMl 

Grolier 


& 

Skinner AIM 


ATC- 

Keesler 


Density 


.62 


.38 


.33 


.39 


.23 


Response/ 


frame 


1.13 


2.16 


1.29 


1.28 


1.57 


Sentences/ 


frame 


1.71 


2.55 


1.25 


2.11 


1.5 


Readability 


8.33 


12.0 


14.9 


7.2 


10.4 



We can compare these programs against each other 
to note the variation in our 4 objective measures. We 
would like to urge caution in drawing any conclu- 
sions at this stage in our analysis. With the limited 
data thus far collected v/e are not certain yet how valid 



these measures are. We are not certain yet what arti- 
facts may be present, what sample size is required, or 
whether programs at various levels have the same cor- 
relations with student data. As mentioned previously, 
these data cannot be used without correlation with after- 
the-fact data such as error rate, final test scores, 
student opinions, etc. These kind of data we have not 
yet collected. 

A readability index appears at this stage to be a reli- 
able and valid measure. If the readability index is say 
12, the program will probably be difficult for 6, 7th 
and 8th graders. Conversely if a college program has 
an index of 6 or 7, the presentation will be aggravatingly 
slow and probably boring to that level of student. The 
sentences per frame do not appear to or indicate much 
at this time. Since sentence lengths may vary consid- 
erably, we also tried “number of lines,” but without 
success. Words also did not appear meaningful. A bet- 
ter measure might be the number of ideas or S-R 
associations, but these are too subjective. Some volume 
measure which correlates with size would be helpful, 
but thus far we have not found one. 

The responses-per-frame measure indicates that, 
on the average, informational frames were not used 
very frequently. This measure does not appear to be 
of significance at this time. Further exploration of the 
reading vs. responding experiments may reveal data 
which will indicate the number of responses which are 
necessary or beneficial. The density ratio measure 
appears to have the most significance at this stage in 
our research. Logically it looks good and theoretically 
it appears sound. Correlations between density and 
student level, error rate, final test score, and retention 
need to be determined. Comparison of several differ- 
ent density ratio programs, say ratios of .20, .40, .60, 
and higher, need to be prepared and tested. If signi- 
ficant correlations are again found, alternate-tract 
programs can be prepared on the basis of density and 
a significant objective measure and programming tool 
will have been discovered. 



SOME GENERAL RULES OF FRAME CONSTRUCTION 

Sfuart Marguiies 

Stuart Marguiies draws a distinction between “critical response’' frames, and what 
he calls “everything” frames. In the critical response situation, the student is assumed 
to know only that material to which “he has responded" correctly. A student reading 
an “everything" frame is held responsible for every item of information in the frame, 
even if it is irrelevant to program objectives. Implications of the critical vs. everything 
distinction are pointed out, particularly as they apply to rules for response cueing. 
Stuart Marguiies contends that the critical response principle is consistent with labo- 
ratory devised principles of learning. 



INTRODUCTION 

The present paper is directed to the identification 
and analysis of some general rules of frame construc- 
tion. These rules allow us to recognize and correct 
the deficiencies in such frame sequences and single 
frames as the ones below: 



1 . 

a. Because of its inejQficient cutting arm, the lay- 
man reaper can perform only 120 operations. 

A total of 360 operations would require 

reapers. 

b. The layman reaper is restricted to 120 opera- 
tions because of inadequacies in its 



c. Because of its cutting arm, only 120 opera- 
tions can be obtained from the 

reaper. 



2 . 

The immediate use of goods in return for the pro- 
mise to pay later is characteristic of a purchase 
made on credit. 

“Fly now, pay later” is an example of a purchase 
made on 



3. 

Cortical convolutions increase as we ascend the 
phylogenetic scale. 

Thus as we ascend the phylogenic scale we 

observe an (increase/decrease) in 

the number of cortical convolutions. 



ANALYSIS OF SIMPLE FRAMES 

Before examining the faults in the frames above, 
let us consider frames 4 and 5 below. These Lwo frames 
illustrate in an exaggerated fashion two contrasting 
philosophies of frame construction. The purpose of 
both frames is to teach the information that the sym- 
bol * is the equivalent of plus. 



4. 

* is the equivalent of plus 
4*3 = 



5. 

* is the equivalent of plus 
4-1-3= 



Both of these frames adhere to generally accepted 
tenets of programming technique. They present small 
units of information, require active responses and may 
be arranged to give immediate reinforcement. Yet 
most people would intuitively select frame 4 as pre- 
ferable to 5; they would agree that frame 5 can teach, 
but would feel that frame 4 can teach better. 

This intuitive preference is amply supported by 
the theoretical arguments of the learning tlieorist. The 
shaping process requires that the experimenter build 
only on established behavior. Frame 4 is satisfactory 
because a correct response requires mastery of the 
material presented; the writer is therefore able in 
later frames to build upon the concept that has been 
taught. Frame 5 must be rejected because a correct 
response need not indicate understanding of the new 
information. 

CRITICAL PROCEDURE VS. EVERYTHING PROCEDURE 

Implicit in the preceding discussion is the following 
principle, which is held here to be the fundamental rule 
of efficient frame construction: 

In each frame, a student can be held responsible only for 
that portion of the presented material to which he has re- 
sponded correctly. Subsequent frames may build only upon 
such material. 

This principle will be referred to as the “critical 
response” or “critical” principle and may be con- 
trasted with a second position, which will be called 
the “everything-is-leamed” or “everything” princi- 
ple. The “everything” principle, of which frame 5 is 
an illustration, may be stated as follows: 

In each frame, a student may be held responsible for all 
material presented whether or not he utilizes this information 
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in making an immediate response. Subsequent frames may 
build upon all material presented. 

An analysis of the following simple frame sequence 
may be useful in illustrating the differences between 
the two contrasting principles. 



a. * is equivalent to plus. 

* was used by the Greek, Thesalius. 

3*4 = 

b. Name the Greek who used the symbol *. 



This frame sequence is consistent with the every- 
thing principle, but violates the critical principle 
because material called for in frame b has not been 
responded to in the preceding frame. Violations of this 
fundamental rule occur quite frequently in the work of 
inexperienced program writers, but are rarely so bla- 
tant as in frames a and b. A more typical example 
was given in the introduction section of this article and 
is reproduced below: 



a. Because of its inefficient cutting arm, the lay- 
man reaper can perform only 120 operations. 

A total of 360 operations would require 

reapers. 

b. The layman reaper is restricted to 120 opera- 
tions because of inadequacies in its 

c. Because of its cutting arm, only 120 operations 

can be obtained from the reaper. 



Again, material called for in frames c and b has 
not been correctly responded to in preceding frames. 

PROGRAMS VS. VEXTBOOKS 

Learning from textbooks is based on much the same 
principles as learning from programs constructed on 
the everything-is-learned rule, because in both cases 
the student is expected to read and retain all presented 
material. The active response required by the every- 
thing program is not very important; the student is 
responsible for material to which he has not made an 
active response. Theoretically, then, a student might 
learn as well by reading such programs with the re- 
sponse blanks filled in as by going through the programs 
and making all the correct responses. 

This fact may partially explain those surprising re- 
search results which indicate that students who read 
through a “program” with the response blanks filled 
in do as well in post-test performance as students who 
go through the program making all the required re- 



sponses. These studies invariably conclude that stu- 
dent responses are not critical to learning; such results 
may actually be attributable to the fact that the pro- 
grams thus studied have utilized everything-is-learned, 
rather than critical-response, procedures. 

In an everything program, there may be no signifi- 
cant difference in the post-test performances of those 
students who are required to respond and of those stu- 
dents who are ^ven no opportunity to respond. In a 
critical program, the student required to respond may 
be expected to achieve better results than the student 
who merely reads. 

In other words, in a critical program, active re- 
sponses are integral to learning, while in an every- 
thing program, they are not. 

COMPARISON OF “EVERYTHING” VS. 

“CRITICAL” program 

The everything-is-learned program can result in sat- 
isfactory post-test performances, just as inefficient 
textbooks may result in learning. However, it is the 
contention of this paper that everything-is-leamed 
procedures result in less motivationally sound and less 
pedagogically efficient learning than do critical-re- 
sponse procedures. Critical-response program se- 
quences employ the laborator}' principle of shaping in 
that they build upon reinforced behavior. Everything- 
is-learned programs, on the other hand, require an 
inefficient expenditure of student time and demand 
study skills which few readers possess. The student 
must painfully read and reread each frame, attempt- 
ing to master any element which might be significant. 
He may discover that he will later be asked about some, 
but not all, of the material which is included in a 
frame. In this event, he may begin to learn selectively. 
And he must decide which information, apart from that 
which is required in the response, is important and 
most likely to be called for in subsequent frames, even 
though, as Mechner has pointed out, the student is 
often new to the subject matter area, and far less com- 
petent than the writer to judge what is important. 

USE OF SUPPORTING MATERIAL 

It may seem that the writer of everything frames 
has greater opportunity to introduce illustrative sup- 
porting material; after all, he counts on the student to 
read, study and retain every element in every frame. 
Paradoxically, the contrary is true. The everything 
writer must often limit himself to a sparse presentation 
because he may otherwise cause the student to devote 
himself to learning material intended only for illustra- 
tive purposes. Consider the following frame, which 
teaches the delivery of material requires 10 days. 
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Sfuarf MarguJfes 



It requires 7 days to pack and assemble material 
after the request is made. 

It requires 3 days to label the material for 
delivery. 

How long is the time between request and 
delivery? 



This frame offers no problem for the critical-re- 
sponse writer who holds the student responsible only 
for learning that delivery takes 10 days. However, 
the everything-is-learned writer who introduces such 
material faces the problem that the student will study 
all the supporting information in preparation for sub- 
sequent frames which may ask how much time is re- 
quired for labeling, or which operation requires 7 days. 
This information may be useful for illustrative pur- 
poses, but unworthy of retention. 

RULE OF CUEING 

According to the critical-response rule a student 
may be held responsible for material to which he has 
responded correctly. It is clear, however, that a cor- 
rect response may indicate only a limited mastery of 
the presented material. This leads to an important 
corollary to the critical-response rule. It is formally 
stated below: 

A student is held responsible only at the minimum level 
of understanding required for a correct response to a frame. 

The practical implication of this rule is that the pro- 
grammer may not assume conceptual understanding 
when the student is responding to extraneous cues. The 
following frame illustrates this danger: 



All Gaul was broken into THREE divisions. 
Thus Gaul consisted of T parts. 



Here the student can respond correctly without really 
reading or understanding any of the material presented 
in the frame. This frame is functionally identical to a 
frame which reads: 



Copy the word THREE 



For this reason, several writers have termed such 
frames as “copy-response frames”. The disadvantage 
of such frames is that they are inefficient. In the ex- 
ample above, the student receives practice only in 
copying the word “three” rather than in learning that 
Gual is divided into three parts. More efficient ways 
of writing the above frame are: 




Gaul was divided into parts occupied by the Celts, 
the Belgae and the Acquitani. 

Into how many parts was Gaul divided? 



(Note that by the critical-response rules given above, 
the reader need not learn the names of the Gallic 
tribes.) 

DANGERS OF INADVERTENT COPY FRAMES 

Learning theorists point out that concept formation is 
dependent upon maximum generalization within a 
class. Copy frames adhere to the anti-thesis of this 
principle because the student reading a copy frame re- 
peats the information presented; he does not sample 
from the wide range of instances which define the class. 

The failure to properly develop conceptual under- 
standing is not limited to frames employing such bla- 
tant cues as T Markel and Mechner have 

pointed out that there are many subtle ways in which 
a frame writer can inadvertently give away the correct 
response and thereby transform what might otherwise 
be a well cued frame into a copy frame. (This topic 
has also received extensive treatment in the field of 
test construction.) The following frame contains an 
example of a subtle giveaway cue: 



The immediate use of goods in rc.tum for the 
promise to pay later is characteristic of a pur- 
chase made on credit. 

“Fly now, pay later” is an example of a purchase 
made on 



Here, the phrase “purchases made on” preceding the 
word “credit” in both sentences clearly gives away the 
desired response, and serves to reduce the frame to 
a copy frame. 

TECHNIQUE FOR APPROPRIATE CUEING 

The relationship between a copy frame and a prop- 
erly cued frame is continuous rather than dichotomous; 
a frame may be more or less cued, and the desired 
response may be more or less given away. A useful 

187 



o 



Trends In Programmed Instruction 

technique which maximizes generalization and serves 
to ensure meaningful utilization of new materials is to 
require that each restatement of a concept utilize new 
terms and, when possible, new perspectives. Consider 
the examples below, each of which attempts to teach 
the concept that cortical convolutions increase as we 
ascend the phylogenetic scale. (It is assumed in these 
examples that aU the pertinent terms are familiar to 
the target population.) 



1 . 

Cortical convolutions increase as we ascend the 
phylogenetic scale. 

As we ascend the phylogenetic scale, cortical con- 
volutions (increase/ decrease) . 



2 . 

Cortical convolutions increase as we ascend the 
phylogenetic scale. 

As we ascend the phylogenetic scale, cortical con- 
volutions become (greater/smaller) . 



3. 

Cortical convolutions increase as we ascend the 
phylogenetic scale. 

As we go from lower organisms to higher organ- 
isms, the number of cortical convolutions becomes 
(smaller/greater) . 



4. 

Cortical convolutions increase as we ascend the 
phylogenetic scale. 

A platypus is a mammal. Which has the greater 
density of brain furrows, a platypus or a duck? 



5. 

Cortical convolutions increase as we ascend the 
phylogenetic scale. 

The diagram below represents cortical sections 
from a lizard, a duck and a rat. Write the name of 
the appropriate organism below each diagram: 



Frame 1 is clearly a copy-response frame, frame 2 
represents a slight improvement because it used the 
words “becomes smaller” instead of the word “de- 
creases.” Similarly, frame 3 is a considerable im- 
provement because many of the terms in its second 
sentence have been changed, while frames 4 and 5 con- 
tain sufficient variation to be considered acceptable. 

SUMMARY AND CONCLUSION 

Critical-response procedures employ laboratory de- 
rived principles of shaping. The student is held re- 
sponsible only for material to which he has responded 
correctly and, as a coroUary, only at the minimum 
level of understanding required for such responses. 
Conceptual understanding is aided by the presentation 
of each restatement of a concept in new terms and, 
when possible, new perspectives. 

In a critical program, responses are, then, an integral 
part of learning. In an everything-is-leamed program, 
they are not, and the student reads and rereads 
frames, attempting to master even unessential illustra- 
tive material. Critical-response principles can be used 
to correct a wide range of deficiencies in frames and 
frame sequences. Programs that employ these prin- 
ciples build upon reinforced behavior, and are consis- 
tent with laboratory derived principles of learning. 





CONSTRUCTION OF AN ACADEMIC PROGRAM FOR 
SCHOOL CONSULTANTS IN PROGRAMMED LEARNING 

Herbert H. Hughes 

Concerned with the mounting evidence that representatives of publishers, industry 
and the military are not communicating effectively with teachers and administrators 
in schools, Herbert Hughes of Colorado State College proposes a program for educat- 
ing programmed learning consultants at the graduate level. 



Issues raised by programmed instruction will even- 
tually be fought in the broader context of learning and 
education, and it is time for leaders in the new move- 
ment to recognize this situation and direct more atten- 
tion to it. The assumption that all programming ex- 
perts have to do is to make materials available, and 
teachers will automatically accept the materials and 
the philosophies undergirding them, may be mislead- 
ing. It must not be forgotten that education has been 
literally full of “panaceas” and “breakthroughs,” many 
of which have been buried in a few short years. 

To insure its lasting influence on education, program- 
med instruction must be merged with learning and edu- 
cation not merely through theoretical work and pro- 
grammed material development, but more importantly 
through frequent personal interaction between profes- 
sionals in programming and teachers in school envir- 
onments. This should be a major concern since it may 
be the prime requisite for survival of programmed 
instruction as a major influence on classroom learning. 

This paper is speciflcally concerned with describing 
plans for an academic program for such consultants, 
and to present a rationale for its development. The 
end-product desired will serve as a point of departure. 
The man to do the job in schools will be a sort of 
middle-man in programmed instruction, part program- 
mer, part teacher, and part psychologist. Regardless of 
his academic emphasis, he must be able to work com- 
fortably with persons from the three specialties. He 
must be an effective educator because his work will be 
in the schools. 

Financing the cost of training a person for these 
competencies must be considered. One possibility is to 
establish cooperative agreements with organizations 
producing programs to be marketed. To optimally and 
efficiently train school personnel in the various phases 
of programmed instruction, many institutions of higher 
learning may have no other choice than to establish 
working agreements with private programming or- 
ganizations. Using the personnel and physical facili- 
ties of these organizations for practicum and intern- 
ship experience makes it financially feasible for the 
institution to provide a well balanced academic se- 
quence in programmed instruction. 



The educational level at which programmed learning 
consultants should be trained must be decided. The 
bases for programmed instruction have been relatively 
sophisticated, and it is recommended that such spe- 
cialized academic training should be restricted to the 
graduate level. Since quantity, flexibility, and quality 
must be considered at the present stage of the develop- 
ment, it has been recommended that such training be 
at the master’s specialist (one year beyond the mas- 
ter’s) and doctoral levels. The master’s candidate 
would be trained to function chiefly as a liaison person 
between school systems and research and program- 
ming organizations. Practicum experience from one to 
three months in organizations developing program- 
med materials would be required at this level of train- 
ing. Two emphases could be pursued at the specialist 
(Ed. S.) level, one in consulting on programmed in- 
struction and the other in programming instructional 
materials. Similar coursework would be required for 
both emphases with the major difference lying in the 
nature and length of the practicum or internship. The 
specialist degree in consulting would be designed to 
provide school systems with a specialist who is quali- 
fied to provide in-service education in programm.ed 
learning, evaluate programmed materials for use in 
the schools, and to consult with teachers using pro- 
grammed materials for instructional purposes. The 
specialist in programming would provide many of the 
same services, but would be especially trained to work 
with school personnel in the actual development and 
programming of instructional materials. Doctoral em- 
phasis in programming would remain primarily at 
the minor or supporting level since there seems to be a 
trend for the professional in programming to remain 
“psychologist programmers” or “educator program- 
mers” rather than to establish separate programmer 
cults. Another possibility is a doctoral major in teach- 
ing and learning technology. As programmed instruc- 
tion, educational television, computer technology, and 
other new developments mature, it is likely that de- 
mands for highly trained personnel knowledgeable in 
all these areas will rapidly increase. 

With a general conception of what kind of consultant 
is envisioned, the problem of what coursework should 
go into the consultant’s academic program must be 
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decided. It appears that the outstanding potential pro- 
grammed instruction may have depends on the degree 
to which professionals in programming concern them- 
selves with the following four points. All are of vital 
concern in planning specific content for a course of 
study in programmed instruction since they help define 
the programmed learning consultant’s functions. (1) 
Those in the everyday business of education must con- 
tinually examine the role of ultimate objectives or real 
life goals and evaluate course content, teaching meth- 
ods, and student performance in the light of progress 
toward ultimate objectives. Ultimate objectives are in 
the final analysis education’s reason for being. It 
appears that programming has encouraged excessive 
dependence upon objective, but short term, criteria 
for learning and education. Many teachers and edu- 
cators also do this, but a conscientious teacher or 
educator has probably more difliculty than others avoid- 
ing questions about ultimate outcomes because he must 
be responsible for the composite end-product of public 
education. Programmers and programmed learning 
consultants must focus more attention on this problem 
in the future. Hopefully, concern about ultimate ob- 
jectives can be taught. Formal courses in educational 
philosophy, curriculum development, and learning and 
personality theory are recommended to accomplish 
the task. (2) A strong background in research method- 
ologj' and measurement seems a must for a program- 
med learning consultant. Evaluation and experimen- 
tation have been an integral part of the process of 
programming from the very beginning. Program- 
ming is experimentation, experimental construction 
of instructional materials. Probably no other educa- 
tional innovation has from its very inception placed so 
much stress on evaluation. Programs are built on the 
basis of how well they meet carefully defined, ob- 
jective goals. Some kind of objective criterion mea- 
sure of the performance of students for whom materials 
are intended is always necessary, even for those who 
view program writing itself as an art. Problems of 
sampling, and defining and constructing criteria are 
crucial, but in addition, new programs are frequently 
expected to justify their usefulness through experi- 
mental evidence demonstrating superiority of pro- 



grams over other materials or methods. The eventual 
acceptance of programming by teachers can be en- 
hanced considerably by researchers wiUing to criti- 
cize and continually evaluate their own work. There- 
fore, even at the master’s level courses in tests and 
measurements, statistics, and research methodology 
are recommended. (3) Methods used by programming 
organizations to disseminate and advertise programs 
must be monitored carefully. The programmed instruc- 
tion movement is now being scrutinized by a wide 
variety of potential users, and a most important cri- 
terion for the movement’s early acceptance will be ad- 
herence to a proper code of ethics by professionals who 
are engaged in dissemination of programs. Ethics 
should be stressed, especially in internship or practicum 
experiences, as well as in measurement and specific 
programming courses for the programmed learning 
consultant. (4) Orienting teachers to programmed in- 
struction rates high priority, but probably the most im- 
portant, as well as most difficult, task for program- 
ming consultants in the schools is to develop ways to 
integrate programmed materials with the wide range 
of existing materials and curricula. Programming skill, 
technical knowledge, and a great deal of educational 
know-how will all be needed to meet the challenge. A 
programmer will have to be particularly skillful in 
adapting his ideas, methods, and techniques to either 
new or old personnel and curricula. An internship or 
practicum is essential for building these skills, and 
experience in the schools would seem highly desirable. 

Programming’s rapid growth and increasing influ- 
ence on education has created the demand for a link 
between program producers and consumers. In par- 
ticular, school systems lack consultant personnel who 
not only understand theories and techniques of pro- 
gramming, but also are competent educators. Now is 
the time for careful thought to be given the academic 
training necessary to prepare these personnel. An in- 
terdisciplinary course of study for school consultants 
in programmed instruction has been presented. A 
strong praciicum or internship provides the base for 
the program which requires at least an exposure to 
most areas in education and psychology and consid- 
erable work in research measurement. 
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PROBLEMS IN THE RELATIONSHIP BETWEEN SUBJECT MAHER AND 
PROGRAMMED LEARNING SPECIALISTS WITH IMPLICATIONS FOR TRAINING AND RESEARCH 

John A. Modrick 

John Modrick points out that programmed learning specialists can make a sig- 
nificant contribution by devising a technology which brings together the subject matter 
and instructional specializations. Subject matter structure and metalanguage are pro- 
posed as new tools for use by the instructional programmer to perform his function. 



In programmed learning we are confronted with an 
ancient and persistent problem arising from the condi- 
tion that teaching requires the mastery of two things: 
a subject matter and the techniques of communication 
associated with the transmission of knowledge. The 
subject matter specialist, be he physicist, sociologist, 
poet, economist or musician, has content in the form 
of ideas, principles, and facts which he wishes to trans- 
mit to his students. The programmer has principles 
and techniques for presenting that content in visual, 
auditory, verbal or performance form. If the instruc- 
tion is to be effective, the subject matter specialist 
must ask: “This is what I want to teach; what do you 
need to know in order to tell me how to present it?” 
The programmer, in turn, must ask: “What is it you 
wish to teach so that I can determine the most efficient 
way to present it?” 

The successful management of the interaction be- 
tween these two specialties or functions is a critical 
activity in teaching. Programmed instruc .on can 
make a major contribution to education by develop- 
ing a technology for this interaction between subject 
matter and instructional techniques. 

The problem of combining subject matter and in- 
structional methodology is not new. One solution has 
been to leave both functions with one individual. Thus, 
the subject matter expert writes textbooks and lec- 
tures. All too often the result is books and lectures 
which, though scholarly, are grossly ineffective for in- 
struction. Another alternative is that the instructional 
methodologist teaches the subject matter. It has been 
claimed that this is the direction of modem professional 
education, and it is further claimed that the outcome 
of this alternative has been teaching which is a model 
of technique but ineffective because the teacher did 
not himself understand what he was trying to communi- 
cate or he had nothing to say. 

This approach of “two skills in one skull” is not 
effective. For most people there is not enough time 
in one life span to acquire and maintain proficiency in 
a subject matter and in instructional techniques as 
well. There is also the possibility that the traits which 
make for excellence in both areas are incompatible or 
occur together infrequently. The effective, widespread 
use of programmed instruction will depend on collab- 



oration between subject matter and programmed learn- 
ing specialists. The programmers will be specialists 
in instructional methods, program writing, and audio- 
visual techniques. They will collaborate with subject 
matter specialists from seveial disciplines of the arts, 
sciences, and vocations. 

A successful collaboration has existed for years in 
the construction of achievement tests. Subject mat- 
ter experts provide the content, that is, identification 
of the specific knowledges and skills to be tested and 
the actual writing of items, while the test experts per- 
form the functions of editing and evaluating the items. 
This collaboration may provide a prototype for the re- 
lationship between the programmer and subject mat- 
ter specialist. However, the model is not adequate for 
two reasons: (1) the test is much shorter than a pro- 
gram and does not present the same problems of the 
management of material, and (2) the items in a test 
are independent of each other; the test builder need not 
be concerned with the sequence of items. In program- 
med instruction the sequence of items is the key to the 
program. 

We will have separation of the two functions. Al- 
though some specialization on types of subject matter 
will undoubtedly occur, the typical programmer will 
I’.ave to operate across many subjects. He must be 
able to communicate with the subject matter specialist 
for the purpose of obtaining the information neces- 
sary to write the specific items of the program and to 
organize the content of the program in a sequence that 
will be effective for learning. 

There is, then, a third function of communication 
between the two specialties for the purpose of analyz- 
ing the subject matter into a form necessary for pro- 
gram writing. We usually leave the development of 
proficiency in this communication to trial and error 
during on-the-job experience. However, we need stand- 
ardized techniques for this analysis. 

To develop these techniques we need two new tools: 
(1) a set of constructs for describing the conceptual 
structure of a subject matter, and (2) a metalanguage 
for talking about subject matters. 

Neither of these ideas is yet clear in my own think- 
ing. However, structure of a subject matter is re- 
flected in the fact that a subject cannot be entered at 
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random but there is an order in which the material 
must be taken. The central ideas in structure are se- 
quence and the condition that some items are prereq- 
uisite to others. Some topics and concepts are ad- 
vanced and sophisticated while others are basic and 
elementary; some topics and concepts must come first 
while others are built on, integrate or are extensions 
of the basic items; some concepts .are definitional 
and existential while others are relational. There are 
similarities between new and previously mastered con- 
cepts which can beJused to advantage in presenting 
new material. 

I tend to think of the metalanguage primarily as vo- 
cabulary, although I am certain that it includes gram- 
matical rules. As you know, the term is used by logi- 
cians to designate a language used to make assertions 
about another language. We can make the oversim- 
plified distinction that the subject matter is an object 
language for talking about phenomena in the real 
world and tlie metalanguage is a language for talking 
about the object language. 

The first step in inventing the metalanguage would 
be the development of a taxonomy of concepts and 
skills. There are two sources of information on which 
to base the taxonomy. One possible source is the func- 
tion of a class of concepts in the subject matter. The 
other source is the psychological characteristics in 
terms of the cognitive and learning processes of the 
individual. Whichever source is used to identify classes 
of concepts, these concept classes are the basic ele- 
ments of the language to which symbols are assigned. 
These symbols are the words of the metalanguage. 
There must be also rules for combining symbols into 
assertions about the object language. The concept 
classes are also the things that go into the subject mat- 
ter structure. 

If you find these ideas a little nebulous, I share the 
experience with you. I cannot give you specific exam- 
ples of subject matter structures or metalanguages be- 
cause I do not know of any. However, let me give 
some examples in an attempt to illustrate what I mean. 

The first example comes from my experience as a 
teacher. In recent years I have taken to 3sking ques- 
tions on the assigned materials instead of lecturing at 
the students. This procedure has been very instructive 
for me. There are two observations I want to make. 
One is that the student tends to learn snatches and 
isolated bits of the material. Although an orderly de- 
velopment and an interdependent sequence of concepts 
are in the subject matter, the student seldom picks 
that up readily. What I have to do, as a teacher, is to 
recapitulate this conceptual organization by a combi- 
nation of questioning, prompting and interpreting while 
the student puts in the isolated bits and snatches which 



he recalls. In this manner, the structure of the subject 
matter is emphasized. 

The second observation is that for more concepts I 
have to backtrack to more fundamental concepts and 
ideas and work the student through the development of 
the concept to be learned. Take the concept of test 
validity as an example. Although the student can learn 
the glossary definition, it has little meaning for him . 
In order to develop the concept of validity, one must 
build on the ideas of a continum of individual differ- 
ences, criterion, correlation and prediction of perform- 
ance from test scores. These concepts are the building 
blocks of the structure of the concept of test validity. 
Disregarding this structure produces instructional 
chaos. 

My second example comes from the idea of tech- 
niques of evaluating an integral in calculus, at least as 
I remember learning it. This is not a diflficut idea, yet 
many people have a difficult time mastering it. How- 
ever, when one looks at the number of concepts and 
techniques which are prerequisite for learning to 
evaluate an integral, the concept is truly impressive. 
Among them are the concepts of function, area, incre- 
ment, summation, approximation, limit, infinitely 
small, addition and multiplication. Usually the student 
is exposed to these concepts at various grades in school. 
These concepts are the content of the structure of this 
complex concept. The ease of learning this concept is 
probably a function of the mastery of these component 
prerequisite concepts. 

My last example deals with electronic maintenance. 
I had the task of preparing material for a program to 
be used in research on training. We wanted to teach 
maintenance procedures on actual military electronic 
equipment to electronically naive subjects. There- 
fore, a program had to be worked out by a psychol- 
ogist who knew very little about the equipment and an 
engineer who knew the equipment so well that much 
of his activity had become automatic. We engaged in 
a lot of groping and trial and error, but after a while 
it became clear that there was an order and system 
in what we had to do. For example, we found three 
major divisions of the material which had to be taken in 
this order: (1) a limited number of fixed checkout 
procedures to be learned with normal indications, (2) 
recognition of negative indications introduced into the 
checkout procedure, and (3) families of localization 
procedures associated with each of the negative indica- 
tions. Further, within these three divisions, it was also 
possible to identify at each step in the sequence 
specific knowledges and skills which the student had to 
master in orderto perform that step. These prereq- 
uisite knowledges and skills are then sorted into re- 
lated groups and taught to the student. Within each 
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of the divisions we found it either necessary or ad- 
vantageous to take items in a particular order. 

As we v/orked through the subject matter we began 
to see its organization and structure and to recognize 
types of concepts. The individuals who did this work 
will be able to apply the method to other program- 
ming tasks, but individuals facing this type of subject 
matter for the first time will have to repeat the same 
trial and error. Science and technology cannot be so 
dependent on individual idiosyncratic skills acquired 
through experience. We must develop standard pro- 
cedures and techniques for the analysis of subject 
matters and this development requires the development 
of the constructs of structure and metalanguage. 

The structure and metalanguage are the tools for 
analysis of subject matter into the kinds of concepts 
being taught and the sequencing of this content into 
an optimum sequence for learning. This is the work of 
the programmer. Programming is not just the writing 
of items or frames. The writing of items is important 
but only part of the job. Programmed instruction is a 
technology for the transmission of knowledges and 
skills to the student. The programmer’s objective must 
be to optimize this communication and to do this he 
must be able to analyze subject matter from a peda- 
gogic point of view. 

The basic idea in the structure is that some items of 
knowledge are prerequisite to the acquisition of ether 
items. There are two ways in which a concept can be 
prerequisite. One is logical necessity. The other form 
is prerequisiteness in terms of ease of learning or un- 
derstanding. There are undoubtedly concepts which 
can be learned directly but, if these concepts are pre- 
ceded by certain kinds of learning, they could be learn- 
ed much more rapidly. We need to identify classes of 
concepts, study the manner in which each class of con- 
cepts is learned, determine techniques to facilitate the 
learning and apply these techniques to the writing of 
programs. 

One can, of course, point out that programs have 
been written in the absence of subject matter structure 
and a metalanguage. However, existing programs are 
most numerous in subjects having very little structure, 
such as language, or subjects having a traditional 
empirically-derived structure, such as mathematics 
and logic. Languages have virtually no hierarchy of 
concepts. There are basic categories of vocabulary and 
rules of grammar which must be mastered, but reach- 
ing the intermediate levels of proficiency consist of in- 
creasing the number of associations within a category 
and increased speed of recognition of words and con- 
struction in various contents. In mathematics, on the 
other hand, a definite hierarchy exists in which more 
complex and difficult concepts are based on simpler 



concepts learned first. However, the existing are 
in mathematics is not necessarily the true structure. 
This is apparent in the successful attempts of recent 
years to teach supposedly advanced topics, such as 
set theory, to high school freshmen. When we have the 
techniques for analyzing the structure of a subject mat- 
ter, we may find that there are many pathways through 
mathematics and we must choose between them in 
terms of some index of effectiveness. 

Research on concept learning has been on a plateau 
for some time. Most experimental work has been a 
variant on the early advances of Heidbreder, Hull, and 
Smoke. However, there are occasional findings that 
point in the direction of a taxonomy of concepts. The 
concept classes are usually related to ease of learning. 
For example, Heidbreder (1946) found that the order 
of increasing difficulty in learning concepts was con- 
crete objects, spatial forms and abstract numbers. 

Kendler and Vineberg (1954) studied the learning of 
compound concepts formed by combining two lower 
order concepts. For example, the subject learned 
a compound concept in which the combined attri- 
butes of size and shape of objects both defined the con- 
cept. The ease of learning these compound concepts 
was related to the number of component concepts (0, 
1, 2) that had been learned prior to learning the com- 
pound concept. 

An interesting incidental finding is that one of the 
concepts was extremely difficult to learn. Four simple 
concepts were used: shape, size, color, and a fourth 
called part-whole. Part-whole is a relational concept 
and it was consistently more difficult to learn. The 
hypothesized explanation is that there is not a natural, 
readily available, verbal-mediating response for the 
part-whole characteristic. There are, however, readily 
available mediators for the other concept attributes. 
In almost any subject matter, many of the concepts 
have no natural, or readily available, verbal media- 
tors to facilitate learning. 

A final example is the work of Bruner, Goodnow, and 
Austin (1956) on conjunctive and disjunctive concepts. 
A conjunctive concept is one in which both of two char- 
acteristics, A and B, must be present; a disjunctive 
concept is one in which one or the other of two char- 
acteristics, A or B, must be present. 

Recently Neisser and Weeiie (1962) reported a study 
of concept learning in which ten types of concepts were 
used. The ten concepts included univariate and multi- 
variate concepts representing various types of con- 
junction and disjunction. The ten types of concepts 
represented three levels of complexity. The levels of 
complexity corresponded to the difficulty of learning 
the concepts and within a level of complexity there 
were differences in difficulty by type of concept. 
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My reason for reviewing these studies is that they 
represent some ways of defining concept classes. The 
classes are defined by formal abstract and general 
properties independent of content. Concept classes so 
defined are related to differences in learning. The next 
step is to apply these types of analyses to concepts 
in subject matter. We should then be able to formulate 
additional rules of programming that will increase 
the effectiveness of programmed instruction. 

My presentation has been almost entirely in terms 
of knowledge and concepts, while skill has been neg- 
lected- This has been largely a convenience in speak- 
ing. Skills are another aspect of the taxonomy and 
structure of a subject matter. My attention has been 
caught recently by a “Taxonomy of Educational Ob- 



jectives” by Bloom. In the cognitive domain the objec- 
tives include such skills as interpretation, application, 
judgment and various kinds of analytical skills. This 
is probably not an adequate taxonomy of skills, but it 
is a start in the right direction. 

In conclusion, I repeat that programmed learning 
can make its greatest contribution by devising a tech- 
nology which brings together tlie subject matter and 
instructional specializations. This will be done if the 
programmer sees himself as a subject matter analyst 
who extracts from the subject matter the information 
necessary' to accomplish the instructional objectives. 
To perform such an analysis, he wiU need new tools 
like the subject matter structure and metalanguage. 



IBEX— A METHOD OF INSTRUCTION 

Robert Pursley, Major, MSC 

Major Robert Pursley is an instructor at the Army’s Medical Field Service School. 
He has the difficult task of teaching relatively dull administrative procedures to rela- 
tively disinterested Medical Service Officers. By combining business simulation, role 
playing and programmed instruction, he presents training that seems to crack the 
motivation shell and challenge the intellectual abilities of his students. 



Glaser (1962) in his introductory comments to his 
recent book entitled, “Training Research and Educa- 
tion” comments; 

One of the essential obligations of a society at peace is the 
education of its members. It is also true that an essential 
mission of a military establishment during peacetime is the 
training of its personnel. 

But, before we can train our military establishment, 
we must keep its members healthy and able to perform 
the “vigour” of this training. The prevention and treat- 
ment of disease and injury, be it ever so slight, is a 
24-hour, 7-day a week business. It is to this purpose, 
that is, the prevention and treatment of disease and 
injury of the Army establishment, that the members of 
the Army Medical Service devote their full-time effort. 

The faculty of the Medical Field Service School must 
train our students. Although we draw completely upon 
the civilian educational institutions to provide us with 
the finest physicians, dentists, and veterinarians, the 
Army must complement their civilian education with 
the pure military skills which are vital to our defense. 
Our newly commissioned Medical, Dental, and Veter- 
inary Corps officers must acquire a basic knowledge of 
militarjr map reading, weapons capabilities, infantry 
and artillery tactics, and the problems of practicing 
military medicine under combat conditions. Since these 
young officers may become the team captains of to- 
morrow, every effort must be made to develop and en- 
hance their leadership abilities. Further, we must 
orient this new officer to the administrative processes 
of the Army establishment. It is to this purpose that I 
wish to relate a recent experience with programmed 
instruction. 

But first I would like to ask you two questions: (1) 
How can we make the learning of the “administrative 
processes” more effective and enjoyable?; and (2) 
How can one generate an interest of physicians, den- 
tists, and veterinarians along the lines of military let- 
ter writing, personnel accounting and reporting 
procedures, or just the everyday routine variety of 
“paperwork”? 

This is one of our motivational problems. I do not 
wish to imply that this routine paperwork is either un- 
necessary or unimportant. On the contrary, good 
administrative processes are vital to good patient care. 



The answers to these questions have been the subject 
of discussion for many hours by the instructors of the 
Medical Field Service School. 

Heretofore, the lecture has been the principal meth- 
od for presenting these subjects. In an effort to 
increase the effectiveness of these hours, we have 
made what we believe to be a unique experiment by 
combining two relatively new methods with one older 
method of instruction. 

The methods used are: (1) Business simulation, 
sometimes called gaming; (2) Programmed instruc- 
tion, primarily branching; and (3) Roleplaying. 

First let me describe the business simulation method. 

“A business simulation or game may be defined as a se- 
quential decision-making exercise structured around a model 
of a business operation, in which participants assume the role 
of managing the simulated operation.” (Greenlaw, 1962) 

At the Medical Field Service School we have designed 
a 2-day classroom “game” which is presently 16 hours 
in duration. Our classroom “game” has been named 
IBEX which is an abbreviation for the title In-Basket 
Exercise. In one sense, this exercise simulates the ad- 
ministration problems which appear in one’s in-basket. 
The 112th General Dispensary located in Garmisch, 
Germany, has been selected for the exercise model or 
hypothetical work environment. 

The transition from normal classroom lectures to the 
IBEX method is not sudden. Great effort is taken to 
set the “mood” wherein the participants can obtain 
definite facts about their new job. The student offi- 
cers are slowly pulled into a hypothetical work situa- 
tion in which they pretend being assigned and perform- 
ing duty at an overseas general dispensary. They are 
asked to go by the assumed name of a Medical Corps 
officer who has just recently arrived in Germany from 
his orientation course at the Medical Field Service 
School at Ft. Sam Houston, Texas. 

At this point, instruction of the “administrative proc- 
esses” is begun. The methods used to introduce teach- 
ing objectives referred to as “plays” include: (1) Let- 
ters, memos; (2) Telephone calls; (3) Roleplayers; 
and (4) Other. 

Fundamental details of a “given subject area” have 
been programmed to prepare the participant to respond 
to a “planned roleplaying situation.” Remember, at 
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this point the participants have received little or no in- 
struction on the “how to’s” of paperwork before read- 
ing their programmed pamphlets. Time is tdlowed in 
the classroom to read the short branch program before 
the roleplaying situations are begun. Also, time is al- 
lowed for the participants to discuss the newly ac- 
quired knowledge with their neighbor. The average 
reading time for the programs used to date is between 
15-20 minutes (small steps). 

The programmed instructional material fills the gap 
between the participant’s lack of Army experience on 
the everyday detailed procedures and his intuitive 
problem-solving capabilities. In other words, a large 
volume of standing operating procedures or procedure 
guides have been greatly reduced to the essentials 
which enable the student to begin functioning in a work 
situation. 

Upon completion of a short discussion period the in- 
structor names a student to participate in a short skit 
wherein he must apply his new knowledge to a deci- 
sion-making situation. Normally, an informed “role- 
player” (usually a faculty member) is used to present 
the problem to the student participant. Once the par- 
ticipant has made his decision and indicated to the 
class his action, the skit is stopped by the instructor. 
The student participant is asked to discuss the good 
and bad points of his solution to the problem presented. 
Once the student roleplayer has discussed his own 
actions, the other members of the class are encour- 
aged to ask questions or make comments. 

Almost without exception, each teaching objective 
is developed without comment by the instructor. This 
permits the instructor to actually observe the success 
or failure (terminal behavior) of the teaching objec- 
tives of the student participant and his fellow students 
asking him the questions. 

Let me discuss the advantages of this method as we 
see them: (1) Variety; (2) Active participants (hot 
passive listeners); (3) Immediate feedback; (4) Prim- 
ary roleplayers are peers; (5) Forces the instructor 
to prepare clearly defined teaching objectives; and 
(6) Teaching objectives are developed by fellow stu- 
dents. 

The following are the disadvantages: (1) Time con- 
suming; (2) Requires a great amount of preparation 
time; (3) Instructors must be thoroughly trained; and 
(4) Minor changes require complete rewrites. 



In conclusion, I would like to summarize the findings 
obtained from a short questionnaire which was de- 
signed to measure the student’s attitude or reaction to 
oar first exercise. We used a 2-page questionnaire 
with completion type items. Some sample questions 
were: (1) How do you feel about IBEX in general? 
(2) How do you feel about the subject material present- 
ed in IBEX? (3) To what degree did IBEX create a 
better learning situation than lectures? 

Generally, all participants indicated that the exer- 
cise was very effective as a training method. Most 
students indicated that the subjects were essential and 
the method of presentation made “what could be dull 
material interesting.” All indicated that they believed 
the exercise was more effective than the lecture 
method. 

A specific question was included to determine which 
method used in IBEX was most meaningful from the 
student’s point of view. The question was: “From 
which of the following methods used in IBEX did you 
learn most: (1) Programmed instruction; (2) Roleplay- 
ing; (3) Committee discussions; (4) Referring to gen- 
eral reference material; (5) Telephone calls; and (6) 
Written communications?” Results were: 51% indi- 
cated roleplaying; 38% indicated programmed instruc- 
tion; and 11% indicated all others. 

The comments made by the students were most en- 
couraging. For example: “More such type instruction 
should be instituted.” “The most enjoyable and inter- 
esting instruction for a subject which is extremely un- 
interesting to an individual who has spent so much time 
in classes during the last 28 years.” “If possible, I 
would suggest even more student participation. If 
one knows he might be called upon, he will more likely 
endeavor to understand the material.” “Use program- 
med instruction more. To me, this is the most bene- 
ficial part of IBEX.” 

From our first exercise, we feel that combining 
simulation, programmed instruction, and roleplaying 
provides another very useful instructional method. 
Further, it is my personal conviction that programmed 
instruction can be a very effective method in generat- 
ing interest of professional groups such as physicians, 
dentists, and veterinarians in purely administrative 
subjects. 
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NOVELTY EFFECTS OF PROGRAMMED INSTRUCTION 

W. James Popham 

W. James Popham snvestigafed the possibility that the effects produced by machine 
presented programmed instruction are due in part to the novelty of the teaching situ- 
ation. He matched two groups of sixth grade, half ov whom worked with teaching 
machines for a fall semester. In the spring, both groups learned the same new mate- 
rial by machine. Both groups showed equivalent gains in knowledge, suggesting that 
novelty effects, if present, were not sufficient to produce differences. 



“How much of the apparent success of teaching ma- 
chine learning is merely due to the ‘novelty effect’ 
produced by the devices?” Particularly in the case of 
teaching machines, in contrast with programmed 
books, one might suspect that the very uniqueness of 
the devices themselves could produce heightened stu- 
dent motivation and hence superior learning. While 
programmed texts are in many respects more com- 
parable to conventional instructional materials, even 
the effectiveness of these materials may be underes- 
timated because of their novel features. 

This question mirrors a concern of educational re- 
searchers as well as school administrators who might 
consider possible acquisition of self-instruction equip- 
ment. To the researcher, the question dictates caution 
in the interpretation of what generally have been rath- 
er encouraging experiments testing the effectiveness 
of programmed learning approaches. To the school 
administrator, the question may engender enough skep- 
ticism to discourage the use of self-instruction devices, 
even on a trial basis. 

Only Porter (1958) reports research bearing on this 
important question. He offers evidence that after a 
five month period of using teaching machines there 
was no indication of lessening student motivation. 
More recently, both Keislar (1959) and Carr (1960) 
have warned against the possible influence of novelty 
effect, an effect which might lose its value over an ex- 
tended period of time. In commenting on the Porter 
research, Carr urges that “. . . more research of this 
sort (dealing with novelty effect) is necessary before 
we can be sure that confirmation can be relied upon 
to maintain its reinforcing property for very long pe- 
riods of time.” 

If the foregoing observations are accurate, it be- 
comes apparent that attention should be given to the 
possible influence of novelty effect upon learning ac- 
quired through approaches featuring auto-instructional 
devices. The experiment to be described was explicitly 
designed to explore this question. 

The experiment proposed to supply evidence relevant 
to the question of how much influence, if any, a postu- 
lated novelty effect would have upon the effectiveness 
of learning in a self-instructional situation. Specifi- 
cally, the null hypothesis tested was the following: 



There will be no significant difference in the achievement 
of two comparable groups of elementary pupils in a self- 
instruction learning situation when one of the groups 
uses teaching machines for the first time and the other 
group has had extended previous experience with the 
machines. 

Essentially, the research design of the project in- 
volved comparisons of the performances of two student 
groups on an identical self-instruction learning task. 
It was postulated that a novelty effect which might be 
conducive to better learning was present to a greater 
extent in the case of the students who were learning 
via machines for the first time. 

In greater detail, one group of sixth grade elemen- 
tary school pupils (hereafter referred to as the low 
novelty effect group) had a semester’s experience with 
a teaching machine approach to learning the fundamen- 
tals of algebra. During the 1961-62 fall semester, each 
student devoted three 30-minute periods a week to an 
algebra program presented by a Foringer .#2002 
teaching machine. The machine instruction took place 
in a vacant classroom, adjacent to the regular class- 
room, under the supervision of a student assistant. 
During this period the second group (hereafter re- 
ferred to as the high novelty effect group) had no 
contact with the teaching machines, but was intro- 
duced to comparable algebra topics by the regular 
classroom teacher. 

During the spring semester both groups devoted three 
30-minute periods per week to an elementary geome- 
try program, also presented by the Foringer devices. 
Students in the high and low novelty effect groups 
were randomly assigned in equal proportions to four 
teaching machine work groups who used the six avail- 
able teaching machines at different times. It was as- 
sumed that the greater familiarity of the low novelty 
effect group with the machines and general proce- 
dures associated with self-instruction would tend to 
mitigate that interest in the geometry materials which 
might be attributable to the newness of the instructional 
approach itself. Whereas, with the high novelty effect 
group, using the teaching machines for the first time, 
the uniqueness and hence inherently interesting nature 
of the situation would be apparent. 
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Subjects in the two groups were drawn from a sin- 
gle sixth grade class in the Frederic Burk School, the 
San Francisco State College elementary demonstration 
school. The two groups were matched at the beginning 
of the school year according to pupils’ scores on the 
California Test of Mental Maturity and SR A Arith- 
metic Achievement Test. Recent scores on both these 
measures were available for all students from school 
records. Twelve students formed the high novelty 
effect group, while eleven students were included in 
the low novelty effect group. 

The criterion measures by which the null hypothesis 
was tested were students’ difference or “gain” scores 
between an initial geometry test, administered at the 
beginning of the spring semester, and two other 
•geometry tests administered later during the spring 
semester. All items for these tests were drawn from 
the suggested tests accompanying the TEMAC pro- 
grammed geometry learning materials. 

The first test, a 40-item instrument, was adminis- 
tered to all subjects during the first week of Februaiy. 
This test, which covered frames 1-200 of the self-m- 
stniction program, was used to assess the knowledge 
of geometry that students possessed prior to their use 
of the machine-presented geometry programs. It 
should be noted that the performance of the two groups 
on the February test was quite similar. For the high 
novelty effect group the mean number right was 6.75 
and the standard deviation was 3.54. For the low 
novelty effect group the mean number right was 7.00 
and the standard deviation was 2.28. 

All students initiated their work on the geometry 
programs on February 13. On March 6, a 63-item 
geometry test covering frames 1-600 of the program 
was administered to both groups. At that time nine 
half-hour work sessions had been made available to 
each student so that, barring absence, each student had 
devoted 270 minutes to the machine-presented geom- 
etry materials. Difference scores of the two groups 
were contrasted by the nonparametric Mann-Whitney 
U Test. Differences between the groups with respect to 
(1) number of frames completed and (2) time devoted 
to the programs were also checked. 

In the original design of the experiment the March 
6 test, administered relatively soon after students had 
initiated their work on the geometry programs, was 
considered important because a possible influence at- 
tributable to novelty effect might be present during 
the early stages of the two groups’ use of the geom- 
etry programs. It was considered possible that such an 
early novelty effect influence might be dissipated as 
the experiment wore on, hence the early testing. If a 
significant performance difference favoring the high 
novelty effect group had been observed during the 



March 6 testing, it was planned to administer periodic 
achievement tests to both groups after every nine or 
ten teaching machine work sessions in order to note the 
continued presence or disappearance of this perform- 
ance differential. 

As will be seen in the results section of this report, 
periodic testing was not dictated by the results of the 
analysis of the March 6 test data, hence only a final 
criterion examination was administered at the close 
of the spring semester on June 4. This terminal exami- 
nation consisted of 116 items with 145 possible points 
for a perfect paper. The test covered frames 1*1850 
in the geometry program. Difference scores of the two 
groups were contrasted by the Mann-Whitney U Test. 
Differences between the two groups were also noted 
with respect to number of frames completed as well as 
number of sessions devoted to the teaching machines. 

Also on June 4 a questionnaire designed to assess 
student attitudes toward the teaching machine mode 
of instruction was administered to both groups. Dif- 
ferences between questionnaire responses of the two 
groups were tested for statistical significance by the 
nonparametric Fisher exact probability test. 

Before turning to the performance of the two groups 
on the criterion tests, it will be useful to examine the 
comparability of the students on the measures of intelli- 
gence and arithmetic achievement used to match the 
two groups. In Table I the means and standard devia- 
tions for both groups on these two measures are pre- 
sented. 

Analysis of March 6 results. Prior to analyzing test 
results *of the two groups after the March 6 hasting, 
differences were appraised between the groups regard- 
ing time expended on the teaching machines as well as 
number of frames completed. 

Because of the possibility of student absences result- 
ing in less machine time for one group, the minutes 

Table I 

Performance of the High Novelty Effect Group and 

Low Novelty Effect Group on the California Test 
of Mental Maturity and the SRA Arithmetic 
Achievement Examination 





California Test of 
Mental Maturity 


SRA Arithmetic 
Achievement 
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Novelty 

Effect 


12 


68.58 


7.95 


57.33 


12.41 


Low 

Novelty 

Effect 


11 


68.64 


7.20 


57.18 


11.27 
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devoted by each student to the programmed materials 
were recorded. Time devoted to the programmed learn- 
ing by each group was quite similar, with mean minutes 
of machine time for the high novelty effect group 
equalling 242.5, and for the low novelty effect group 
233.0. The number of frames completed ^ each 
group was almost identical (Xine = 256.16; Xine = 
261.40). 

The mean of the two groups’ “gain” scores were 
practically equivalent. The mean of the high novelty 
effect group was 21.00 while the mean of the low 
novelty effect group was 21.80. The difference be- 
tween the two groups, as indicated by a Mann-Whitney 
test, was not statistically significant (t/ = 55.5). 

Analysis of June 4 results. Since the March 6 test 
revealed no superiority for the high novelty effect 
group, the terminal examination administered on June 4 
was the only other criterion measure employed. 

The analysis pattern was similar to that employed 
for March 6 data. First, differences between the two 
groups with respect to time devoted to the program 
and number of frames completed were inspected. As 
in the March analysis, the differences between the 
groups on both of these scores were trivial. The mean 
number of half-hour work sessions widi the machines 
by the high novelty effect group was 33.92. For the 
low novelty effect group the mean number of sessions 
with the machines was 32.18. The mean number of 
frames completed by the high novelty effect group was 
998.8, while for the low novelty effect the mean num- 
ber of frames completed was 1,026.5. 

The mean performance of the two groups in terms 
of “gain” scores was quite similar. The mean of the 
high novelty effect group was 41.42 and the mean of 
the low novelty effect group was 39.73. The difference 
between the two groups, when analyzed by a Mann- 



W. James Popham 

Whitney test, was not statistically significant (U = 
62.5). 

On the basis of data analysis for \ o+h criterion tests, 
then, there were no significant differences between the 
two groups. Hence, the null hypothesis U'.ider investi- 
gation was considered tenable. 

Results of the three geometry quizzes administered in 
February, March, and June are reported for the high 
novelty effect and low novelty effect groups in Table II. 

Analysis of questionnaire results. Both groups com- 
pleted a questionnaire at the close of the experiment 
which solicited student opinions regarding the teaching 
machine mode of instruction. In an effort to induce 
greater validity in the responses of the students, the 
questionnaire directions implied that responses to the 
instrument would be forwarded to each pupil’s new 
junior high school in the faU, where they might play 
a role in determining whether the student would use 
teaching machines in the future. Students were asked 

(1) Whether they liked to use teaching machines; 

(2) Whether they would prefer to study mathematics 
by teaching machines next year; (3) Whether they felt 
the teaching machines were used too often in class; 
(4) Whether they would have gotten more or less out 
of the course if the teaching machines had not been 
used; (5) Whether they became tired of using the ma- 
chines as the school year progressed; and (6) Whether 
they would choose to use teaching machines for part 
of their study in their next school. 

Responses of the two groups to all of these questions 
were almost identical. ITie lack of a statistically signifi- 
cant difference was verified by computing six Fisher 
exact probability tests. 

Consideration of the results of this experiment leads 
to the conclusion that insofar as novelty effect was 
produced (for both groups) and diminished (for the 



Table II 

Performance of Twelve High Novelty Effect Pupils and Eleven* Low Novelty Effect Pupils on 

Three Geometry Examinations 
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8.86 


45.82 
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38.82 
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* One low novelty effect student was absent during the March 
testing. 
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low novelty effect group) by the design of the study, 
it was not a factor of importance in the self-instruc- 
tion learning task. The null hypothesis was accepted 
on the basis of the criterion test administered in March, 
a few weeks after the high novelty effect group started 
to use the teaching machines. The null hypothesis was 
also accepted on the basis of the criterion test admin- 
istered four months after the high novelty effect group 
initiated their teaching machine work. 

It should be pointed out that the "gain” scores used 
to test the null hypothesis do not actually represent 
advancement on the same test, for on each testing occa- 
sion a geometry examination was devised which v/ould 
cover only that portion of the program which might 
reasonably have been encountered by the students in 
the experiment. Thus, a longer test was used for the 
March testing than had been used in February, and a 
still longer test was used in June. 

A comment should also be made regarding the rea- 
ron for the selection of algebra and geometry materials 
for use with sixth grade pupils. These programs were 
chosen since it v/as assumed that pupils’ knowledge of 
both subject matter areas would be relatively minimal. 
The geometry program was selected to follow the alge- 
bra program used by the low novelty effect group since, 
although it contains concepts and skills which are dif- 
ferent from these encountered in algebra, it falls in the 
same overall content field of mathematics. It was be- 
lieved that if a program from a totally different field 
(such as a foreign language) had been chosen for the 
second semester, the low novelty effect group might 
have viewed the new learning materials as being suffi- 



ciently novel to cancel out the novelty effect differential 
the study was designed to produce. 

It should also be noted that since the experiment was 
conducted in a college demonstration school where col- 
lege student-observers in classrooms are fairly common, 
the possible influence of “hawthome” effect was prob- 
ably diminished. On the other hand, this same situa- 
tion might make it more difficult to generate a novelty 
effect for sixth grade youngsters who have experienced 
several years of adult observation. 

Finally, it must be emphasized that while the results 
of this study will be viewed with favor by those pro- 
ponents of programmed instruction who believe the 
merits of this approach to be based on sound learning 
principles rather than novelty effect, the results of a 
single study are far from conclusive. We must await 
results of further experimentation before the novelty 
effect variable can be excluded from our considerations 
in programmed learning research. 

The performance of two groups of sixth grade young- 
sters on an identical teaching machine learning task 
was contrasted in this experiment. One group of pupils 
had been exposed to a teaching machine learning task 
for a semester, the other group used teaching machines 
for the first time. It was posbilated that “novelty effect” 
would be present to a greater degree in the case of 
those youngsters in the latter group. On the basis of 
two criterion tests, one administered early in the experi- 
ment and one administered several months later, no 
difference between the two groups was discovered. 
Novelty effect was concluded to be of no importance 
in the learning situation. 



A POTPOURRI OF PROGRAMMING TECHNOLOGY 

James L. Evans 

James Evans presents some new ways to prompt the behavior of o programmer os 
well os a student. The programmer can prompt his production of frame materials by 
one of two permutation techniques. A modified binary fiow chart, developed to 
organize Latin translation functioned os a very effective prompt when given to students 
using the final program. By using appropriate directions, students can be prompted 
into making organized, permanent notes of programmed materials. Finally, a toil-first 
vanishing technique is reported that teaches simple cartooning. 



The programming enterprise is a serious effort to 
apply behavioral science to the process of human learn- 
ing. An easily overlooked fact is that the programmer 
himself is a behaving organism, and is subject to the 
same laws of behavior which he is utilizing in his pro- 
gramming technology. As my colleagues and I have 
attempted to apply behavioral principles both to our 
students and to ourselves in practical programming 
situations, a number of techniques and by-products 
have emerged. My paper is a sample of some of these 
by-products. 

In programming, we attempt to arrange a series of 
stimuli which insure that the student will proceed 
rapidly and effectively to mastery performance. We 
acknowledge the fact that the behavior of the student 
is controlled by the stimuli we present him. Can we 
then deny that our own behavior in programming is 
largely controlled by the stimuli impinging upon us? 
Let me quote from a description of how one scientist 
took self-prompting seriously. 

“He took 63 cards and placed on them the names and 
properties of the elements. These cards he pinned on the wall 
of his laboratory. Then he carefully re-examined the data. 
He sorted out the similar elements and pinned their cards 
together again on the walls. A striking relationship was thus 
made clear. He now arranged the elements in seven groups. 
... He continued to arrange the remainder of the elements. 
When his list was completed, he noticed a most remarkable 
order. How beautifully the elements fitted into their places!” 

The scientist in question was, of course, Mendeleeff, 
and the result of his self-prompting was the discovery 
of the Periodic Law of Chemistry (Jaffe, 1956). If 
Mendeleeff was wise enough to take self-prompting 
seriously, can we who are in the business of producing 
and controlling verbal behavior afford to take it less 
systematically? I think not. In 1959, Lloyd Homme, 
Robert Glaser, and I described a “verbal matrix” tech- 
nique for use in the generation of programs. Further 
analyses have led us to the conclusions that the two- 
dimensional matrix therein described is but a special 
case of a very powerful and general method of using 
permutational techniques in generating, prompting, and 
analyzing verbal behavior. 

Let me relate an incident which may clarify my pre- 
vious remarks. In reviewing a rather bad program on 



Metric Measurement which turned up on my desk one 
day, I was struck by a singular fact. Apparently the 
programmer was attempting to establish certain rela- 
tionships wi thin and between certain classes of terms 
which appeared in the program, and he was approach- 
ing this in a quite unsystematic and ineffective manner. 
I then began examining each firame of the program to 
find a term which might represent a member of a larger 
set. For example, the term “meter stick” is obviously 
a member of the set of measuring instruments, and I 
immediately examined the remainder of the program to 
see what other measuring instruments were mentioned. 
Interestingly enough, at no point in the program was 
mention made of instruments for measuring time, al- 
though tim e measurements are fundamental to any 
measuring system. 

Figure I represents my expanded analysis of the 
topic, which I have chosen to call the “set-permuta- 
tional” technique. Several properties of the classes 
themselves are of interest. In some cases, the members 
of the set are ordered, for example; there is an obvmus 
ordering of metric prefixes. With respect to measuring 
instruments, however, no such ordering is apparent.. 
There are some varieties of relationships both within 
sets and between sets. For example, in the metric 
prefixes there are a number of relationships, based on 
factors of ten, to be learned relating prefixes such as 
pico-, nano-, micro-, and so on. Also, within classes, 
there are relationships between basic units such as feet 
and seconds, and derivatives such as feet-per-second 
squared, or acceleration. There are also one-to-one, 
one-to-many, many-to-one, and many-to-many relation- 
ships between classes. For example, the metric pre- 
fixes permute completely with metric suflSxes. That is, 
if you select eleven metric prefixes and five metric 
suffixes, you can generate 55 metric units, such as milli- 
seconds, kilograms, megaliters and so on. Examples of 
one-to-one set relations are between metric suffixes and 
English imits. The student who masters the program 
mi^t be expected to convert from inches to meters, 
liters to quarts, and so on. The fact that length mea- 
surements can range all the way from picometers to 
gigometers also is a powerful prompt to the program- 
mer to point out that different measuring instruments 
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are appropriate, depending on the level of measure- 
ment. For example, a micrometer would be used for 
making measurements in millimeters, whereas a meter 
stick might be more appropriate in making measure- 
ments in decameters. The point is not that the pro- 
grammers must incorporate all these relationships in 
his program. What is important, however, is that 
through permuting within and between the various 
classes, the programmer at least prompts himself to 
consider all of the logical possibilities, and he can select 
examples systematically without a resulting loss in 
generality. 

After the members of two or more sets have been 
generated, the problem still remains of some systematic 
method of generating all the logical possibilities result- 
ing from permuting the sets. In general, two somewhat 
different techniques have proved useful in exhausting 
the logical possibilities resulting from the permutation 
of two or more sets. Suppose the programmer has gen- 
erated a set of subjects, a set of verbs, and a set of 
direct objects for use in generating examples in a lan- 
guage program. In particular, suppose the set of sub- 
jects contains two members (numbered 0 and 1), the 
set of verbs contains five members (0 through 4), and 
the set of direct objects contains three membera (0 
through 2). The more general, and more mundane, 
permutational procedure is to use the odometer tech- 
nique. Using this technique, the programmer will 
proceed from 000 to 001, 002, then advance to 010, 
oil, 012, and so on. In this case, the programmer 
will generate all logically possible example sentences 
before they are exhausted and he begins recycling. An 
alternate procedure can be used when the numbers in 
each set are all relatively prime to each other. In our 



example, since 2, 5, and 3 are all relatively prime, the 
second or relative prime technique can be used. It 
should be noted that by the insertion of one or more 
dummy or blank cards to a given set, the set can 
usually be made relatively prime quite easily. When 
the dummy or blank cards come up, that combination 
is simply ignored. As an example, in conjunction 
with generating all logically possible three-letter words 
of the consonant-vowel-consonant pattern for use in 
a reading program, we had 21 consonants for the first 
position, five vowels for the second position, and 21 
consonants for the third position. By adding one blank 
to one of the piles of 21, we increased its number to 
22, which is relatively prime both to 5 and to 21. To 
use the relative prime method of permuting classes, 
simply transfer a card from the top to the bottom of 
each set, and then record or use the resulting permuta- 
tion. As long as the sets are relatively prime pair-wise, 
all logical possibilities will be generated automatically, 
with no duplications until the possibilities have been 
exhausted and the combinations begin recycling. I 
have an idiosyncratic preference for the relative prime 
technique, if for no other reason than that it seems 
to move faster and does have the desirable feature of 
providing a premixed set of permutations for use as 
examples. 

It might be noted in passing that the use of such 
systematic techniques in example generation in pro- 
grams can be an effective control device over the be- 
havior of the programmer. In analyzing a rough draft 
of our Spanish program, we found quite dispropor- 
tionate utilization of certain nouns, verbs, and adjec- 
tives, although in programming, we had attempted to 
provide equal practice on vocabulary items. In a Ger- 
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man program we are now revising, the relative prime 
technique will be used to assure much more homo- 
geneous practice on individual vocabulary items. 

In summary, then, as an analytical tool, key terms 
or concepts in a subject matter area can be identified, 
the set to which they belong also identified, and ad- 
ditional members or sets generated as necessary. The 
pair-wise permutation of these sets can prompt the 
programmer both in the analysis of the subject matter 
and in the generation of examples. Both an odometer 
and a relative prime technique have been described 
for exhausting the possibilities of the permutation of 
two or more sets. 

The set permutational technique previously described 
has proved to be an extremely useful self-control and 
prompting device on certain phases of programming. 
Unfortunately, however, permutations per se do not 
necessarily result in the ordering or sequencing neces- 
sary in arranging the series of stimulus events which 
constitute a program. As I have described in a pre- 
vious paper (Evans, 1961) sequences or chains of 
behavior, particularly when expressed in a notational 
system borrowed freely from computer programming, 
can be powerful analytic devices and may show inter- 
esting insomorphic relationships even between diverse 
subject matter areas. Behavioral analysis using flow 
chart notation proved to be such a powerful analytical 
tool that we, of course, behaved as any teacher or 
priest possessed of new knowledge would; that is, we 
kqpt the flow charts carefully concealed from our stu- 
dents. When someone outside our programming ranks 
asked the obvious question. “Why don’t you let the 
student see your flow chart?” we became rather ruefully 
aware that we were on the student’s side and there was 
no reason why the student should not see the flow 
charts. By using the program to control the student’s 
use of the flow chart, we discovered a number of in- 
teresting points. First, we were sometimes able to 
achieve a saving of 50 percent or more in the munber 
of frames needed to produce criterion behavior. Sec- 
ond, since the flow charts essentially mediated between 
the initial stimulus and the terminal response, they 
started dropping out of the student’s chain of responses, 
as any operant conditioner who has ever worked with 
behavioral chains in animals would confidently predict. 

Figure 2 shows an e}q)erimental flow chart used m 
teaching Latin translations. We have data from a 
cumulative record of a student translating a series of 
Latin sentences of a particular pattern into English, 
using the flow chart. The data indicate that the initial 
latencies in completing the translation of each sentence 
are relatively long, with successively less time being 



used in translation, cuiminating in a steady rate of 
translation. In the construction of a Bridge program 
in which flow charts for opening bids, responses, open- 
ing leads, and so on, are freely used, it is a routine 
observation that the use of the flow chart “drops out” 
or “short circuits” to the point that the student is no 
longer dependent upon them. As Tom Gilbert (1962) 
would point out, the dropping out of such mediators 
is to be e;q)ected in the attempts of the organism to 
economize his efforts in reducing the delay of terminal 
reinforcement. 

In both our own testing laboratory and field testing 
situations, we had often observed students taking notes 
in either more or less organized fashion, and we had 
often heard the requests of students for advice on what 
sort of note-taking procedure to follow. Again, the 
hangover from conventional instruction was so potent 
that we kept forgetting that we were on the student’s 
side. Eventually, it dawned upon us that there was 
no reason why we could not control the student’s note- 
taking process with a “programmed notebook” with 
the same precision with which we controlled his learn- 
ing with programmed instruction. That is, we wojfld 
make up the “ideal” notebook in advance, and then 
so structure the programmed instruction phase that the 
student would gradually complete the notebook in a 
pre-determined order. In this way, we were sure that 
the student ended up with a comprehensive and care- 
fully organized set of notes, in contra-distinction to the 
often erratic notes taken by most students. 

Figure 3 will give you an idea of how the pro- 
grammed notebook works. After testing a program in 
Matter and Chemical Change, we gave 12 students an 
attitude questionnaire which included, among other 
things, a question on their attitude toward the pro- 
grammed notebook. The students did report the use- 
fulness of the programmed notebook in preparing for 
a retention test. 

The last ingredient of our potpomri belongs in the 
“For What It’s Worth Department.” A much neglected 
area in the field of programmed learning is tiiat of 
programming the learning of skills such as sketching, 
drawing, or cartooning. I would like to offer for your 
attention a device which, through the straight-forward 
application of behavioral principles, has produced sur- 
prisingly good results in teaching even young children 
to produce cartoon figures. The production of a car- 
toon figure by an artist is a clear-cut example of a 
behavioral chain. To teach an artist to run off the 
same behavioral chain, the principle of chaining dic- 
tates that the novice should first provide that detail of 
the drawing which the cartoonist provides last. This 
procedure, of comse, runs contrary to the well-estab- 
lished procedure in which the student is instructed. 
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TO THE STUDENT 

To help you take notes on the material you are learn- 
ing, a special Programmed Notebook has been pre- 
pared for you. As you go through your program, you 
will see important formulas and statements enclosed in 
boxes, like this: 



LOG MN = LOG M + LOG N 




The box is a signal that you are to copy the enclosed 
material into your Programmed Notebook. The num- 



0 



ber in the circle (1.21) tells you where in the 
Notebook to copy this statement. 



When you have completed your program, you will 
have a complete set of notes which you have constructed 
yourself to help you in reviewing the material you have 
learned. 



Figure 3 



“If you’re gomg to draw a bunny rabbit, first draw two 
circles like this.” The device consists of a number of 
acetate sheets bound together in book fashion. V/hen 
the book is closed, the student sees the complete car- 
toon figure. The student is given a Magic Marker and 
instructions to “trace what is in red on each page, and 
then complete the rest of the drawing.” On the sec- 
ond sheet, the cartoon figure is again complete, but the 
tan, which is the last detail of the drawing chain, is in 
red. By using a different color to control the student’s 
tracing behavior, we can force the observing response 
to that portion of the figure that is going to be removed 
in the next step. This technique arose when early tests 
indicated that students were not reliably identifymg 
which links of the chain were being successively re- 
moved. However, by instructions to “trace what is in 
red and then complete the drawing”, we found that 
the observing response necessary for the completion of 
the chain can be easily controlled. At the second step, 
the student would trace the tail, and again would have 
a complete dog before him. He then turns the page, 
and now the tail has disappeared completely, and the 



nose is in red, mdicating that the student should trace 
the nose and then add the tail to complete the drawing 
again. On the next step, the details of one eye are 
traced, and then the student adds the nose and the tail. 
On the next step, the student traces the details of one 
eye, adds the details of the other eye, the nose, and 
the tail. Successive steps fade away the feet so that the 
student provides, in turn, feet, eyes, nose, and tail. Ear 
and head details are then faded away, with the student 
running off the complete chain m each case. In the 
next to the last step, the student traces the outline of 
the body, and then runs off a very well-established 
chain which completes the bunny. The final step in the 
program is simply a blank sheet of acetate, and the 
student produces the complete cartoon figure without 
prompting. The slight discrepancies between the final 
production and the original model are known as artistic 
creativity. Informal observation indicates that the 
younger the organism, the better he chains. Young 
children run through the chrtining sequence beautifully; 
sophisticated adults, however, have learned that you 
begin things at the beginning and are prone to attach 
the tail first, or to violate the law of chaining in other 
ways. 

A minor technical note in the construction of this 
cartoon chaining device is that the artist starts at the 
back of the booklet and works on the backside of the 
acetate. Since the student proceeds through the chain- 
ing sequence from the front, his marks are on the 
opposite side of the acetate from those of the artist. 
The student’s drawings, therefore, can be wiped clean 
so that the program can be reused. 

In this paper, I have attempted to present some 
technological by-products which my colleagues and I 
have found useful in producing programs and in at- 
tempting to make programs more effective. I hope that 
one or more of the techniques which I have presented 
will be of use to you in your own progr ammin g activity. 
In summary, if a potpourri can be summarized, we are 
finding that the programs we are now writing and the 
techniques which we are now using don’t look very 
much like the programs and techniques we were writing 
and using five years ago, three years ago, or even six 
months ago. This is as it should be. This is only an 
indication that we are still on the threshold in learning 
to apply the principles of behavior to the learning 
process. 
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JUDGING THE TEACHING EFFECTIVENESS OF PROGRAMS 

Henry C. Eilis 

Infernal and exfernal characferisfics by which fhe effecfiveness of a program can be 
judged are confrasfed. Predicfion of a sfudenf performance criferion based on Infernal 
characferisfics of a program is nof very reliable. Reliance should be placed more on 
such exfernal measures as: final achievemenf, crude gain, residual gain and rafio of 
gain fo fofai possible gain. 



A relatively convenient way to judge the teaching 
ejffectiveness of programs is to begin by distinguishing 
between internal and external characteristics of a pro- 
gram. This distinction has been proposed by Joint 
Committee on Programmed Instruction and Teaching 
Machines (1963) and appeared in their interim report 
of 1962. Internal characteristics of a program refer 
to features which can be judged by visual inspection 
of the program, whereas external characteristics refer 
to features which are revealed by the performance of 
students after completion of a program or are revealed 
by relatively “objective” features such as data on try- 
outs and revisions, sources of program content, qualifi- 
cations of authors, etc. This paper will deal primarily 
with external characteristics, and in particular with 
problems associated with obtaining and assessing data 
indicating gain in achievement produced by the use 
of a program. 

At the present stage of programming technology, it 
seems desirable to rely somewhat more on external 
rather than internal characteristics ('-x a program in 
judging its teaching effectiveness. It is not an easy task 
to judge the effectiveness of a program by mere inspec- 
tion of it since the viewer may not be entirely sure 
about those features which he should examine. In 
addition, not enough evidence is known about pro- 
gramming technology to permit one to rely entirely on 
internal characteristics, nor can we predict with cer- 
tainty that a program which has certain specified in- 
ternal characteristics will necessarily be a program 
which reflects large gains in achievement. For example, 
a program may have frequent review, appropriate stim- 
ulus control, proper sequencing, and yet show very 
little evidence for gain in learning. Similarly, it is pos- 
sible that a program will appear to have face validity 
and again show little evidence for gain in learning. 
Finally, it should be stressed that no attempt is being 
made to diminish the importance of internal character- 
istics but only to call attention to their limitations. 
Future research in programming technology may reveal 
those internal characteristics which correlate highly 
with whatever is desired of a program. MeaLwhile, 



external characteristics appear more useful as criteria 
for judging programs. 

Two external characteristics useful in evaluating the 
teaching effectiveness of programs will be examined: 
measures of gain in achievement, and evaluation of the 
program by students, teachers and subject-matter- 
experts. This is not an exhaustive list of external char- 
acteristics; however, these two characteristics are per- 
haps those most important in judging teaching effective- 
ness of programs. 

The most fundamental kind of data which reflects 
the teaching effectiveness of programs is some measure 
of gain in achievement. “Does the program actually 
teach what it says it teaches?” is a basic question which 
the potential buyer wants answered. Regardless of all 
the niceties of a program, if it fails to teach it is useless, 
and if it teaches and is accurate in its content, it is an 
extremely useful program. There are several types of 
measures that one may employ in measuring achieve- 
ment. First, one may employ some measure of final 
status in achievement following completion of a pro- 
gram. Second, one may employ crude gain, which is 
simply the algebraic difference between a pre-test of 
achievement and a post-test of achievement. Third, 
measures of residual gain may be determined. Finally, 
one may employ a measure which is the ratio of crude 
gain to total possible gain. Each measure has certain 
properties which will be examined in more detail. 

Final Status Measures. Measures of final status indi- 
cate achievement following the completion of a pro- 
gram. An obvious limitation of final status as a mea- 
sure of achievement is apparent in that although the 
achievement may be relatively high, the measure does 
not directly reflect how much achievement is associated 
with the program per se. Final status measures are 
useful for assessing relative level of achievement but 
they do not reflect that portion of learning which can 
be readily attributed to the use of the program. 

Crude Gain. Crude gain is a superior criterion of 
learning for the purpose of evaluating teaching effec- 
tiveness of programs. If no other instruction is given 
during the pre-post test interval, it is fairly reasonable 
to attribute gains in achievement to the program itself. 
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Except under highly controlled experimental conditions, 
the program is likely to be used in conjunction with 
on-going classroom instructional activities which may 
parallel and duplicate the content and objectives of a 
program. Under these circumstances, gain in achieve- 
ment will reflect both the effectiveness of the program 
and the effectiveness of classroom instruction and it 
will be difficult, if not impossible, to attribute such gain 
to either type of instruction or to know what weights 
to attach to either. If one is to use gain scores as indi- 
cators of the teaching effectiveness of programs, then 
the program, ideally, should be used in the absence of 
additional instruction which is similar to that provided 
by the program. That the latter circumstance may be 
unlikely suggests clear-cut limitations of the gain score. 

Although I have argued elsewhere (Ellis, 1962) that 
the gain score appears to be the most useful in evaluat- 
ing the teaching effectiveness of programs, it has ob- 
vious limitations. The fact that greater gain is achieved 
with one program as compared with another may 
simp ly mean that the former program was adminis- 
tered under conditions where extensive classroom in- 
struction of similar content and objectives were given. 

This limitation of the gain score may lead some to 
conclude that comparative studies provide an answer. 
Comparative studies are those in which some experi- 
mental group, which receives a program, is pitted 
against a control group, which does not receive a pro- 
gram, but does receive “conventional” instruction. A 
number of researchers (Ellis, 1962; Holland, 1961; 
Stolurow, 1962) have pointed out the limitations of 
the comparative or control experimental study. Yet, 
recently the Technical Committee of the NSPI (1962) 
proposed, in a checklist for selecting programs, that the 
program be evaluated against alternate methods of in- 
struction. I have previously described the difficulties 
associated with the comparative study (Ellis, 1962) 
and have described its “non-analytic” character. 

Residual Gcdn. A second limitation of the gain score 
has been described by DuBois and Manning (1957). 
They point out that gain measures are composed of 
two uncorrelated parts, that part which is uncorrelated 
with initial status and that part which is perfectly pre- 
dicted by initial status. Since gain is a composite mea- 
sure, they conclude that a measure of learning which is 
independent of initial status would be superior to that 
of gain which is confoimded by what the learner 
already brings to the situation. As a solution, they 
propose that residual gain be employed, since this pro- 
vides a measure which is independent of initial in- 
dividual differences in learning. Following their line of 
reasoning, it would appear that crude gain, as a mea- 
sure of the teaching effectiveness of a program, has an 
obvious limitation — that of being a composite measure 



which reflects not only the effect of some treatment 
variable, but also the initial status of the learner. 
Although this is a valid criticism of the gain measure 
as an appropriate dependent variable in studies of 
learning, it does not appear to be a sufficiently serious 
critique to warrant abandoning this measure entirely in 
studies of programmed instruction. One may stiU 
evaluate the teaching effectiveness of a program using 
gain providing one takes into account this feature of 
the gain measure. 

Gain/Total Possible Gain. Another way of measur- 
ing gain in achievement associated with the use of pro- 
grammed instruction is to compute the ratio of gain 
actually achieved as compared with total possible gain. 
The obvious advantage of this measure is that it pro- 
vides a way of estimating the efficiency of learning by 
controlling for differences in the initial status of learn- 
ing. For example, if the pre-test measurement of 
achievement is high, then there is little room for pos- 
sible gain in achievement. Gain scores may be mis- 
representative and may appear unusually small because 
of high initial scores. The ratio of gain to total pos- 
sible gain, however, takes into account how much can 
be learned from the program and provides an objective 
index of the program’s efficiency. 

The second external characteristic which is useful 
in measuring the teaching effectiveness of programs 
consists of evaluation of the program by students, 
teachers, and subject-matter-experts. Although data 
on gain in achievement is extremely valuable, educa- 
tors have the ultimate decision of judging whether the 
program is congruent with their own educational objec- 
tives. At GPTC we have had the experience of seeing 
a program lauded by one school system because it 
provided the kinds of skills necessary for students, 
whereas the same program has been rejected or ignored 
by another school system because the program did not 
fit in with the latter’s objectives. In both instances, 
objective data on achievement was essentially the same. 
Although a program may show considerable gain in 
achievement, the educator still has to answer the ques- 
tion, “Is this desirable or worthwhile for my students 
to learn?” Thus it seems quite clear that considerable 
responsibility is placed on educators or other potential 
users of programs to determine if the program con- 
tributes to their own needs and objectives in education. 
The impact of programmed instruction is to require 
greater not less effort and responsibility on the part of 
the user of programs. 

Teacher Evaluation. The teacher must answer at 
least two fundamental questions in order to judge the 
effectiveness of a program: “Do students learn?” and 
“Does the program contribute to my instructional ob- 
jectives?” The first question is relatively easy to answer; 
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at least it poses fewer problems than does the second 
question. An answer to the second question requires 
that the teacher have a rather clear conception of his 
educational objectives. Where the teacher is not clear 
as to his own objectives, he will experience difficulty 
in judging whether a program contributes to his pro- 
fessional task. 

Student Evaluation. It is important that student re- 
sponse to a program be positive. Fortunately, it is 
imlikely that students will enjoy a program if they fail 
to demonstrate at least acceptable amounts of learning. 
Preliminary findings from field-testing of programs at 
Roanoke, Va., indicate that student evaluation of pro- 
grams is positively correlated with the amount of learn- 
ing which occurs. Obtaining student evaluations of 
programs is relatively simple as several questionnaires 
are readily available for the potential user of programs. 



Henry C. Ellis 

Subject-Matter-Expert Evaluation. Finally, the ef- 
fectiveness of programs can be judged by subject- 
matter-experts. The subject-matter-expert can best 
judge the adequacy and correctness of the content as 
distinguished from whether or not students will learn 
from the program. It is important that the user of pro- 
grams have assurance that the program was developed 
with the assistance of a subject-matter-expert prior to 
the publication of the program. 

This paper has described two general ways of 
evaluating the teaching effectiveness of programs: 
measures of achievement gains and evaluation of the 
program by students, teachers and subject-matter- 
experts. Advantages and limitations associated with 
various types of gain measures were described. Finally, 
the necessity of obtaining evaluations in addition to 
learning measures was emphasized. 



VALIDATION DATA FOR PROGRAMMED TEXTS: A CHECKLIST FOR EVALUATION OF TESTING 

Hanneiore Vanderschmidt 

Hannelore Vanderschmidt points out that with programmed texts, the burden of 
student performance is shifted from the student or teacher to the program and its 
publisher. The author then lists and analyzes eight criteria by which both publisher 
and potential user may evaluate the quality of programmed texts and suggests that it 
is the responsibility of publishers to conduct and publish such test information. 



When Harold Sands of the American Institute for 
Research recently addressed a group of Boston pub- 
lishers, he suggested that publishers would have to as- 
sume a much greater degree of responsibility for pro- 
grammed texts than they have for traditional texts. If 
students do not learn from an ordinary text, some of 
the onus could always be shifted to the teacher or to 
the student’s performance. With programmed instruc- 
tion the burden of student performance falls squarely 
on the program and, by reflection, on the publisher. 

The publisher can m^e sure that the program works 
by conducting a thorough field test or having such a 
test conducted by the author or by a professional test- 
ing group. If the publisher prints a validation state- 
ment, including test results, with the program he does 
a service for all of its potential users. By printing the 
validation the publisher also does himself a service. 
For instance, if Mrs. Jones comes to him with the sad 
story that the modem arithmetic program did not work 
with Harold, aged 7, the publisher can point to the 
printed validation data which states that the program 
was meant for students with a 6th grade reading level 
and that it assumed certain arithmetic skills as pre- 
requisite knowledge. 

In general, a validation statement traces the history 
of the program through its research, development and 
field test stages and it defines the program’s educational 
intent, indicating with what group and under what 
conditions the program has been shown to work. 

In order to guide our own company with the test- 
ing and validating of programs, I examined 40 com- 
merically published programs. These were published 
by 15 different book companies and programming 
firms. Of the 40 programs examined, 25 presented 
no validation information at all. Only one program 
contained a validation which leaves no pertinent ques- 
tions unanswered. I rated the validation information 
in the programs according to a list of 8 criteria of my 
own devising. These are: 

1. A statement of prerequisite knowledge. 

2. A statement of terminal objectives. 

3. A pre-test and a post-test with analysis of pre- 
post test data. 

4. A description of the test population. 

5. A statement of error rate. 



6. Suggestions for program administration. 

7. A statement of average time required to com- 
plete the program. 

8. A measurement of student attitude. 

Naturally, the 8 criteria are not of equal importance. 
Those criteria which assess how well the program ful- 
fills its stated claims, 1-4, are, in my opinion, more im- 
portant than the other criteria. On this scale the mean 
rating for validations in the test group of programs was 
1.4 with the “validated” programs taken by themselves 
scoring 3.5. These figures are self-evident. The Amer- 
ican program publishers today are not printing com- 
plete validation statements. 

The Field Test 

Since validation information inevitably harkens back 
to program testing procedures, a good validation neces- 
sarily requires a carefully constructed field test. Such 
a field test typically includes a pre-test which is used 
to select the experimental population, the administra- 
tion of the program to the experimental group, a post- 
test to assess whether the program meets it stated ob- 
jectives and, finally, an evaluation of the test results. 

A fairly simple test such as the following will ex- 
emplify these points. In an arithmetic program for 
grade 1 which I wrote under the guidance of Douglas 
Porter at Harvard University in 1961, we assumed that 
the student could form the numbers 1-10 and could 
match the numbers with the corresponding number of 
objects. Our pre-test of 30 items included 6 items to 
assess whether the student mastered this prerequisite 
knowledge; the remaining 24 items tested his knowl- 
edge of the terminal objectives. Students were selected 
for the experiment if they answered the prerequisite 
questions correctly, but did not answer most of the 
objective questions correctly. Age and I.Q. variables 
were also taken into consideration. The program was 
administered and subsequently a post-test, which was 
identical to the pre-test. Student gains from pre-test 
to post-test were assessed by simple statistical tech- 
niques. 

Prerequisite Knowledge 

To design a field test the experimenter must be able 
to list prerequisites and terminal objectives. Why 
should this material be included in the validation? The 












potential user must know what knowledge is assumed 
by the program to be able to determine whether or not 
his own students have the prerequisite knowledge. If 
not, he can bring the students to the necessary base 
line. None of the programs I examined included a list 
of prerequisites, although one did specify the reading 
level required. I think that a statement of reading 
level is particularly valuable to the user as most pro- 
grams today rely very heavily on the reading knowl- 
edge of the student. 

I am well aware of the difficulties of spelling out the 
prerequisite knowledge. Over-specificity needs to be 
avoided as much as the vague amorphous answer. I 
would hope to see a statement of prerequisite knowl- 
edge less vague than “all that has gone before” or “for 
the 8th grade level,” but, at the same time, more easily 
read than a list of 100 technical terms and 300 con- 
cepts, each of which should be mastered by the 
student. 

Terminal Objectives 

Tests included in a program are often good measures 
of its terminal objectives. Of the programs examined, 
8 contained tests: “diagnostic tests,” “unit tests,” “mas- 
tery tests,” or “pre-post tests.” By scrutinizing the 
tests included ‘n a particular program, a teacher or 
other potential user can gain a fair understanding of 
its terminal objectives. The table of contents of many 
programs also gives some indication of their objectives. 

A more formal way of specifying terminal objectives 
is to list these in operational terms. By this I mean 
that the program lists operations the student can per- 
form on completion of the program: he can spell the 
following list of words, he can define the following 
terms, and so on. The use of operational terms cir- 
cumvents vague descriptive statements such as “he 
appreciates,” “he understands” or “he knows.” As 
John Blyth has pointed out in his excellent article, 
“Behaviorism Is Not Enough,” (1962) there are some 
human activities which cannot be specified in behavioral 
terms. Still, the operational attempt to translate the 
learning activity into a group of measurable skills 
makes those skills readily accessible to testing. To 
return to programmed instruction, unless we can test 
to see if a given terminal objective is actually in 
the student’s repertoire at the completion of a program, 
we have little assurance that the program has fulfilled 
this objective. 

Pre-Post Tests and Pre-Post Test Data 
Pre-post tests should be published with the program, 
not only because of their usefulness as an indicator of 
terminal objectives, but also because by including them, 
one gives the potential user a way to evaluate their con- 
struction and to see whether they constitute a fair 
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measure of the program, if he is knowledgeable. Tests 
are useful for the teacher’s use in ascertaining how well 
his particular students are learning. None of the pro- 
grams I looked at included an analysis of pre-post test 
data. I was surprised to find this because how well the 
students do in the post-test (if it is carefully con- 
structed) is an excellent indication of how well the 
program works. By comparing the pre-post test per- 
formance of the experimental group or groups you can 
determine how extensively and with what groups a pro- 
gram works. I do not advocate statistical raptures. 
These serve to offend the expert and cause the mathe- 
matical layman to flee. But, I would like to see some 
evidence that the test administrator, be he author, editor 
or testing expert, tried to ascertain how well the pro- 
gram meets its objectives. 

I have heard that “you can prove anything with sta- 
tistics” over and over again. I know it is fairly easy to 
“rig” the test and the test results. StiU, most educators 
will try to write valid and reliable tests and to evaluate 
them honestly. 

Test Population 

All of the 15 validated programs in the test group 
contained information about the test population such 
as age, grade level, I.Q., where the tests took place and 
how many individuals were involved. Age and I.Q. 
variables are particiularly important, I believe, because 
they set certain necessary restrictions on the use of the 
program. 

Error. Rate 

The question of error rate has dogged programmed 
instruction since Skinner suggested that it be kept below 
10%. Since then, it has been noted that the student 
could miss all the criterion frames and still have a very 
low rate if the program included a large percentage 
of extraneous responses or responses which were ex- 
traneously cued. Of course, what the student misses is 
more important than how much he misses. Yet, infor- 
mation about the error rate, how it varies from bright 
to slow populations, what has been done to decrease it, 
is pertinent. If nothing else, it attests to the care that 
has gone into developing the program. 

Program Administration 

Suggestions for program administration are very use- 
ful to the teacher, especially if he is among the large 
group who has not used programmed instruction before. 
Susan M. Markle’s Teacher’s Manual for the program; 
WORDS, contains much information which would be 
useful to a teacher. She discusses branching and double 
tracking sequence for different ability groups. She 
suggests whether and how students should use these 
sequences, whether students should be kept together, 
and so forth. I think many programs would be more 











Trends In Programmed Insfrucfion 

effectively used if the author-programmer would follow 
this example and suggest administration procedures. 

Time Requirements 

This information, available from the field test, should 
be included in a validation to help teachers make time 
allotment decisions. 

Student Attitude 

Although I have not read any studies offering con- 
clusive proof that a positive correlation of student atti- 
tude and student achievement exists in programmed 
instruction (1963), I am convinced from experience 
that attitude is a factor in student success. At any rate 
it is a factor in determining whether or not a student 
completes a program. He can hardly be expected to 
learn what it purports to teach if he finds it too dull or 
too punishing to complete. 

Because of these considerations some measure of 
student attitude is valuable. The results of a sophisti- 






catedly conceived attitudinal questionnaire or, in the 
case of an “optional” program, some measure of the 
number of students who finished the program as com- 
pared to the number who started it would give the 
would-be-user an indication of how successful the pro- 
gram was in motivating its students. 

Two publishers have gone on record to say that each 
of their programs will be field-tested and that printed 
validations will accompany each progra n. Theoretical 
considerations for testing and validating programs are 
at this time being studied by a distinguished group of 
scholars working for the Educational Testing Service. 
These efforts are much to be lauded and will, I hope, 
yield a crop of first-class programs. Nonetheless, the 
major effort in assuring quality control of programs 
must come from each programmer and each program 
publisher. Only by producing programs of high quality 
can we make certain that programmed instruction will 
assume a significant role in American education. 
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REAPPRAISAL OF ACHIEVEMENT MEASURES 



Michael A. Zaccaria and John Olsen, Captain, USAF 

Michael A. Zaccaria and John Olsen define present concepts of aptitude and 
achievement reliability and validity and then analyze the merits of test item difficulty 
discrimination developed for aptitude tests but misapplied in achievement testing. The 
authors conclude that achievement tests must measure students against performance 
standards and not compare students relatively among themselves. 



In government and industrial technical training to- 
day we hear a great deal about minimum training to 
meet minimum job requirements. Training managers 
seem to agree that money and manpower are wasted 
by over or under training. They go to great lengths 
in developing curriculum to fit this training philos- 
ophy. 

Unfortunately, once training gets under way we often 
find that the philosophy of minimum training in mini- 
mum time gets lost in measurement policies which are 
based on traditional academic considerations. Too 
often students are measured to fractions of a percent- 
age point against other students without ever being 
measured against minimum job requirements. There 
are two main reasons for this phenomenon. First 
training objectives are seldom stated in definitive 
enough terms and secondly, a relative rather than an 
absolute measurement system is employed. 

To fully understand this complex problem it is 
necessar> to contrast aptitude and achievement testing 
with regard to validity and reliability. It is also neces- 
sary to understand item difficulty and item validity as 
applied in technical traimng. The following terms then 
will be discussed to show the contrast between aca- 
demic and training definitions. 

Validity. How well docs the test measure that for 
which it was intended? In aptitude testing validity is 
used to predict how well individuals will succeed in 
training. Thus, the concept of predictive validity. In 
achievement testing validity is used to determine how 
well the student has achieved the goals of the course, 
thus the concept of content validity. 

Normal Curve. In aptitude testing we wish to show 
a maximum of individual differences. Thus, we like 
to have a normal curve with a maximum spread of 
scores. By use of such a device it is fairly easy to se- 
lect individuals for course attendance. 

How is a normal curve attained? By using a maxi- 
mum number of test questions which half of the exami- 
nees answer correctly. This includes a small proportion 
of very easy questions and a small proportion of very 
difficult questions. 

Item Validity. There are two types of test validity; 
internal validity and external validity. Internal validity 
has to do with consistence of items with the total test 
score. For example, suppose we have scores on a test 



ranging from 50 to 100. Arbitrarily divide the higher 
scoring papers at 90 and better and the lower scoring 
papers at 60 and poorer. If 70% of high scoring indi- 
viduals and 20% of low scoring individuals got ques- 
tion one correct, then this item is consistent with the 
total score. If these proportions were reversed, this 
question would be adversely selective of “good stu- 
dents” or “high aptitudes.” In aptitude testing all 
questions should be positively selective and all adverse- 
ly selective questions must be discarded from the test. 
This should not be the case in achivement testing for 
a number of obvious reasons. 

In a training situation the ideal is for every student 
to get zero on each question of the pre-test before 
training, and for each student to answer every ques- 
tion correctly on the post-test. This would yield a 
range of 0 and a mean of 100. We should not be 
teaching students things that they already know and 
we should attempt to teach all students everything they 
should know. If this ideal tr ainin g goal were complete- 
ly reached, we would not have a normal curve of 
achievement scores. We would have everyone scoring 
100% on the final examination. Consider the prob- 
lems this would cause for the training administrator 
or academician. 

External item validity is a good concept for both 
aptitude and achievement tests, under certain condi- 
tions. The problem of using an external criteria, such 
as job success, for achievement test items is that there 
are too many factors other than job competence that 
determine supervisor ratings of job success. In a recent 
study, for example, it was found that supervisors gave 
graduates high performance scores on tasks which the 
graduates had never performed. Individual personality 
of graduate and supervisor, work conditions, and doz- 
ens of other uncontrolled variables make up what is 
referred to as “job success.” There are some goals we 
can foresee and can test, but this does not satisfy us. 
We want a normal grade spread. 

If all students answer a question correctly under 
existing systems that question gets thrown out, or it 
is made harder by making it trickier. This should not 
be the case. If, for example, everyone completing basic 
training should know that in event of capture in time 
of war he should only give his name, rank, serial num- 
ber, and date of birth, then this should be a test item. 
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If 100% of students answer this question correctly, 
then we have accomplished this objective. Removing 
this question from the test is ridiculous. The achieve- 
ment test should measure how well we have achieved 
our training objectives. If such a question is removed 
from the test, we will not know if students in subse- 
quent classes have learned this objective. If the in- 
structor knows students in past classes have achieved 
so weU on this item that they are no longer tested on 
it, he may feel it is unnecessary to even discuss the 
subject in class. 

There is no reason why we should insist on having 
a normal distribution of scores in our achievement 
tests. And yet, this is what we get when we discard 
questions that all students answer correctly. Thus, we 
do not test on objectives that instructors have been 
completely successful in teaching. On the other hand, 
if the instructors do a poor job of teaching the material, 
the question is also discarded because it does not help 
us put scores into a normal distribution. In short, 
these practices do not improve the training situation. 

The normal curve that v/e get from our present test 
validation procedures does not help us to evaluate and 
improve our training. It only enables us to tell which 
are our best students and our worst students, and to 
more or less rank-order them. We then count on 
normal ability to adjust to work conditions. Our pro- 
cedures do insure a higher relationship between apti- 
tude and achievement. Of course, this is built in so 
that our achievement tests tend to become measures 
of aptitude rather than achievement. As a result of 
present procedures, it is very difficult to find out 
whether or not our training goals have been achieved. 
Without this determination there is little chance of 
improving training programs. 

hem Difficulty. Items that are of moderate difficulty, 
those that 50% of the examinees answer correctly, are 
those that maximize individual differences. Test ques- 
tions that are either too easy or too difficult will not 
help give us a normal distribution curve of total scores. 

When a test is to be used for selection purposes, it 
is important that there is somewhat of a normal distri- 
bution curve. 

In aptitude testing, it is perfectly legitimate to make 
easy questions more difficult and difficult questions 
easier. Thus, the end result is a maximum of moder- 



ately difficult questions, and a small number of easier 
and harder questions. This will help obtain a normal 
distribution if that is the desired result of the measure- 
ment program. 

Reliability. Reliability of tests can be divided into 
three areas for the purpose of this discussion: 

a. Equivalence of two tests 

b. Stability of the same test given at two different 
times 

c. Homogeneity or sameness throughout or within a 
given test 

In aptitude testing, aU of the above criteria should 
be met. That is, if two forms of a test are to be used, 
they should be equivalent. The score of an individual 
should be the same whether he takes Form A or Form 
B of the test. If an individual does not receive essen- 
tially the same score today and a month from now on 
the same test, it is of little use for predictive purposes. 
In aptitude testing, it is legitimate to throw questions 
out of a test because they do not contribute to the 
total score for that test. 

For achievement testing, under present measurement 
philosophy, test reliability is only important from the 
standpoint of equivalence of two or more forms of a 
test (except when a course is used as a selection tool). 
All forms of a particular achievement measure must 
be equivalent in that an individual should be equally 
successful regardless of which form of the test is ad- 
ministered. 

In summarizing the differences outlined in the above 
definitions it is fair to say that much of the item 
analysis procedure that is now in use is worse than 
useless for achievement testing in technical and military 
training courses. This type of procedure precludes the 
measuring of objective achievement and improving 
training. Training managers should not require the 
use of item difficulty and item validity (of the internal 
consistence type) as criteria for discarding or retaining 
test questions in achievement tests. 

We in training and perhaps in education should be 
working toward tests that tell us (and the students) 
how much has been achieved during a class or course 
and not merely how the students rank with one another. 
If a solution to this problem cannot be found, we can 
just as well give grades based on shoe or hat size and 
forget the measurement mumbo jumbo we now use. 
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PROGRAMMING THINKING 



Francis A. Cartier 

Francis Cartier points out that a number of aspects of rational thinking (e.g., formal 
logic) can be taught by present programming techniques. Formal logic accounts for 
only a small part of the rational and creative processes we call thinking. An experi- 
mental program on recognition of fallacious reasoning has been partly successful and 
is now being revised. Many other behavioral manifestations of rational and creative 
thinking can be programmed, such as the principle of deferred judgment, the habit 
of considering many alternative solutions, discrimination skills, etc. There is reason 
to believe that programmed instruction need not be based solely on an operant- 
conditioning philosophy of the human mind. All the principles of programming, except 
those of the overt response and immediate reinforcement, could be applied within a 
phenomenological frame of reference, or within a motivational, gestaltist, or even 
existentialist frame of reference. Perhaps the future of programming theory will include 
workable hypotheses which recognize the validity of subjective experience, indeter- 
minacy, and even the inherent ambiguity and covertness of certain mental processes. 



One absolutely sure-fire method of starting an argu- 
ment is to raise the subject of teaching students to 
think with a program. Some of the stricter behaviorists 
refuse to even discuss the subject of thinking. They 
will only discuss observable behavior and will have 
nothing to do with such ambiguous and covert proc- 
esses as thinkin g. So they are likely to reject the idea 
that you can teach thinking wifli programming. 

Oddly enough, the critics of programming take a 
similar position. One of die most common complaints 
we hear from teachers, who hold to traditional meth- 
ods of instruction, is that programming cannot teach 
thinking, and especially fliat it cannot teach creative 
thinking. 

I’d like to take issue with both groups. I’d like to 
report on an experiment in teaching reasoning with a 
linear program; discuss some thoughts on program- 
ming creative thinking; and finally, make some rather 
broad, tentative generalizations about the applicability 
of progr amming theory to mental processes. My sole 
purpose in all this is to provide some thinking about 
programming thinking. 

We can begin by disposing of one of the simpler 
questions: can programming teach formal logic? The 
answer is certainly, “Yes.” As formal logic is usually 
taught, it consists of a very explicit set of rules, very 
much like algebra. Indeed, formal logic is often re- 
duced to an algebraic (e.g.. Boolean) notation system, 
and programs already exist for teaching certain aspects 
of symbolic logic. The most avid critic of program- 
ming would have to admit that, to this extent anyway, 
you can program what he would willingly agree to call 
a thinking process. 



But, with rare exceptions, formal logic is useful only 
to determine the validity of certain kinds of arguments, 
ex post facto. As Chester Barnard says in The Func- 
tions of the Executive, logic is what the executive uses 
to justify the decisions he has already made. 

Few people would claim that formal logic accounts 
for all die higher mental activities. Indeed, it has only 
very little to do with our most important reasoning 
processes. Nevertheless, formal logic describes at least 
one aspect of thinking, and we know it can be pro- 
grammed. 

Let us turn our attention to another aspect. Most 
books on effective thinking devote a chapter or two 
to fallacies of reasoning. Some of these are logical 
fallacies, sometimes called formal fallacies, in which 
the rules of formal lo^c are violated. But others are 
outside the realm of formal logic, as in the use of 
name-calling to refute a point, or the common trick 
of attempting to divert the argument by accusing an 
accuser of a similar transgression. Can programmed 
instruction be used to teach students to recognize these 
kinds of false arguments? Would we call this thinking? 

Take the second question first. Suppose we ask a 
student to read an argument that he has never seen 
before. Suppose we then ask the student whether 
the reasoning in that paragraph is sound or fallacious 
and he can tell us. Wouldn’t you admit, m order 
to do that, the student would have to think? Most 
people would. But suppose further, if he says it is 
fallacious, we then require him to tell us what spe- 
cific kind of fallacy the paragraph contains. Is there 
anyone who will deny that a student would have to 
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think in order to give the correct answer? Let me give 
you an example of what I mean. Here is Frame 126 
from a linear program on recognition of fallacious 
reasoning. 



“Why go to college? Do you expect to make a lot 
of money as a result? Education is no use to you 
unless you learn how to use it. Four years of 
business experience is better than four years of 
isolation from the real world.” 

This is an example of 



The student must decide whether to write in the words 
“sound reasoning^*, or to identify the nature of the 
fallacy. Notice that this is not a matter of formal 
logic. There is no syllogism here. Nor are there any 
direct conclusions drawn. Nor can you deny the truth 
of any of the statements. Yet, it seems to be an argu- 
ment against going to college, and was almost certainly 
intended to sound that way. It is a “ghost” argument. 

Altogether, the experimental program we developed 
at Headquarters Air Force ROTC takes up twelve va- 
rieties of fallacy, using a total of 162 frames. It takes 
about two hours for a college junior to go through it, 
with an average error rate of 5%, and a reasonably 
good score on the criterion test. Of sixteen criterion 
items at the end of the program, items without confir- 
mations, four out of 21 subjects got perfect scores, 
and 12 out of 21 missed two or less. The mean was 
12.8 out of 16, and the median was above 14. These 
are obviously not the scores of a highly successful 
program, but I take them as evidence that the program 
is teaching something, so I am still working on it. 

You may be interested in the way the program is 
developed. After a few introductory frames, the stu- 
dent is taught to recognize the first kind of fallacy, the 
fallacy called the “complex question.” This takes 
twelve frames. After that, the program begins to teach 
the second type of fallacy, the “argument ad hominem.” 
This takes eight frames. At the end of that sequence, 
the student is taught to expect an example of either 
the first or second type. Three or four frames are used 
here to be certain he differentiates between them. Then 
the program proceeds with a series of frames on the 
“ghost argument.” After that sequence, the student 
encounters a few frames which review the three kinds 
of fallacies he has learned. Each example is new. With 
a few rare exceptions in which we take two or three 
frames to examine a particular example, every para- 
graph the student is asked to evaluate is different. We 
are not teaching him to recognize particular fallacies, 
but to recognize the pattern of thinking that produces 
each kind of fallacy. For example, frame 77 reads: 



“There are good books and bad books. The good 
ones should be required reading. The bad ones 
should be burned.” 

This is a * 



And frame 78 reads: 



“The Air Force is concerned with aerospace war- 
fare. The Army is concerned with ground war- 
fare,” is an example of a * 



The response in both cases is, “false dichotomy.” As 
the program proceeds, the examples become more dif- 
ficult and require much more sophisticated evaluation 
processes. Frame 109: 



“Don’t tell me that flourine cannot harm you, even 
in one part to a million of drinking water. Every- 
one knows it is poisonous. If it is a poison in large 
quantities, if cannot be healthy for you just be- 
cause it is taken in small quantities.” 

This is an example of * 



Beginning with frame 98, the student is cautioned 
that there will occasionally be a frame in which the 
reasoning is quite valid. Thus, from that point on, he 
must always consider the possibility that there is no 
fallacy in the frame. These frames were perhaps the 
most difficult to write. I’m happy to say, however, that 
a student occasionally misses one of them. I take this 
to mean that these frames without fallacies are not too 
obvious. 

Like most neophyte programmers, we made some 
sophomoric mistakes at first, but we learned some very 
significant lessons from them. Sometimes this seems 
to be the only way to learn certain things, reinforcement 
theory notwithstanding. 

The mistake we made was to construct our first 
criterion test for this program in the form of a politi- 
cal speech containing a great many fallacies of reason- 
ing. We asked the test subjects to identify the line 
number in which they found a fallacy and identify its 
type. They couldn’t do it. Of course they couldn’t! 
We hadn’t taught them to find fallacies, but only to 
identify them in single paragraphs. So, the next step 
is to prepare additional frames which will develop this 
particular skill. Once they know what they are lookmg 
for, we shall teach them to find fallacies in mock 
campaign speeches, court testimonies, editorials, letters 
to the editor, etc. 

After that, or perhaps at the same time, we shall 
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probably have to spend some classroom time in oral 
exercises in which the instructor will read a speech, 
or the students will listen to a tape-recorded argument, 
and, hopefully, establish some transfer of training to 
the listening situation. There is some possibility that 
this activity might be progranunable, too. We hope 
to get around to trying it eventually. 

From there, it wiQ presumably be desirable to de- 
vise activities in which the students listen to actual 
radio or television speeches or debates in real time, 
read their daily papers, and so on, and search out 
valid and invalid reasoning from what we might call 
“the real world.” 

But will we actually be teaching thinking? Well, 
even though we can show that students do poorly on 
these items before taking the program, and do fairly 
well at them afterward, we should be careful about 
saying whether the program is actually teaching them 
to tiiink. Perhaps all we are accomplishing is getting 
them to think. Perhaps all the program does is get 
them to read more carefully than before, so they 
perceive what a paragraph actually says instead of 
what it seems to say. Whatever it is the students are 
doing, they do it better after the program. And it is 
thinking according to every definition of the term I 
know of. 

Now let us shift our attention to another aspect of 
thinking. The Air Force ROTC has been teaching 
something called Creative Problem Solving since 1953. 
The design for these lessons, the methods of teaching, 
and so on, have been my responsibility since about 
1956, so the teaching of creative thinking is a matter 
of considerable interest to me. I have studied and 
taught it for some time now and am pretty much con- 
vinced that you cannot program creative thinking. You 
can’t program it, in my opinion, because there isn’t 
any such thing. There is only thinking. But real think- 
ing is so rare, that when we see an instance of it, we 
feel we need to celebrate the occasion with the use of 
a special adjective. That’s a greatly oversimplified 
statement of my view, but it will do for the moment. 

Now, obviously, if you want to program something 
according to the principles of operant conditioning, that 
something must take the form of behavior. You cannot 
reinforce a thought, because you don’t know what it 
is. Only when the student provides you with some 
behavioral clue as to the nature of his thought, can 
you apply the techniques of operant conditioning. Usu- 
ally, that behavior takes the form of a written word 
of some kind. 

Teaching creative thinking seems to require that 
we elicit unique responses, unusual responses. What 
we want to reinforce cannot be stated in advance as 
a specific behavioral goal because what we want to 



reinforce is the unexpected, the invention. This is a 
terribly complex problem if you fail to think creatively 
about it yourself. Once you free yourself from precon- 
ceptions and biases, however, it doesn’t look quite so 
formidable. For one thing, in the early stages of a 
creative thinking program, we ought to be satisfied to 
reinforce any behavior that even comes close to being 
creative. We can use the principle of successive ap- 
proximation in several ways here. 

For example, one of the kinds of behavior that 
seems to promote creative problem solving is the con- 
sideration of a large number of alternative solutions, 
instead of considering only one or two. This kind of 
behavior can be defined and recognized and therefore 
it can be programmed. 

Even more rudimentary to good creative problem 
solving is the habit of gathering as much information 
as possible about the nature of the problem before 
attempting to solve it. This seems very obvious, but 
it is not instinctive or automatic and must be learned. 
This habit of restricting the temptation to seek solutions 
until the problem is well understood manifests itself 
as a distinctive kind of behavior, which can be pro- 
grammed. 

Another behavioral manifestation of creative thought 
can be seen in the ability to perceive fine distinctions 
among things that are similar but not identical, and 
another is the tendency to make these fine discrimina- 
tions when analyzing a problem. We know we can 
program the ability to make some very subtle discrim- 
inations because — ^to cite only one example — David 
Klaus has done it with the program in art apprecia- 
tion. 

There are other approximative behaviors we can 
program using techniques now in the programmers’ 
repertoire. A lot can be done in that direction. But 
we cannot ignore the ultimate problem of reinforcing 
the truly inventive response. How do we do that? At 
the present state of the art, I suspect we may have to 
use a live teacher who can decide in each case what 
to reinforce and what not to reinforce. I have no 
statistics on it, but I can tell you about my subjective 
experience in teaching classes in creative thinking. I 
believe it can be demonstrated that the instructor’s re- 
inforcement of creative or semi-creative responses in 
these classes tends to increase both the quality and the 
quantity of creative responses. For such an important 
objective as creative thinking, we should be willing to 
go to the expense of hiring creative teachers to do this 
job. Perhaps, by using programs to teach the more 
prosaic subjects, teachers will have time to do this 
important task. We may have to use highly inefficient, 
subjective, old-fashioned ways of teaching teachers to 
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do this, but that doesn’t worry me much. It has been 
done in the past and can be done in the future. 

In fact, I’d like to suggest that some of the most 
important aspects of creative thinking are necessarily 
subjective and cannot be effectively approached with 
a behavioral philosophy of learning. 

There has been a lot of research recently into the 
psychology of creativity, and it is very confusing. Most 
of it is inconclusive and even contradictory. The num- 
ber of definitions alone staggers the imagination. But 
if you take the trouble to examine it closely, I think 
you may agree with me that the research fails to prove 
conclusively that there are any identifiable character- 
istics or traits that are possessed by the people we call 
creative, and that the rest of us lack. On the contrary, 
the evidence seems to indicate that these so-called per- 
ceptive, creative people lack certain characteristics 
and traits that can be found in comparatively imper- 
ceptive, uncreative people. To put it very simply, most 
of us are crippled by various mental disabilities and 
blocks that prevent us from achieving the clear, un- 
inhibited type of thinking that earns the complimentary 
adjective “creative.” 

So perhaps the best approach to eliciting so-called 
“creative” thought from a student consists in applying 
techniques which will free him from hindrances to 
normal, healthy thinking, rather than providing him 
with some new and unusual way of thinking. 

Obviously, this hypothesis, for I do not pretend that 
it is anything more than that, puts one in the business 
of clinical psychology, or at least on its borders. And 
this is a business in which behavioral psychology has 
been of relatively little help, so far. I don’t mean to 
imply that reinforcement theory is never used in the 
clinic. I have successfully used reinforcement tech- 
niques myself working with functional articulatory dis- 
orders in the speech clinic. But these techniques don’t 
help us much with the client who is disturbed. The 
learning laboratories haven’t elucidated that subject 
much, probably because pigeons don’t get the kind 
of emotional disorders that human beings are heir to. 
Fortunately, however, there is nothing that chains either 
the clinician or the teacher to behaviorism. As teach- 
ers, we are perfectly free to go to phenomenological 
theories, gestalt theories, social psychology theories, 
and even to psycho-analytic theories if they can offer 
anything to help. The big question is, then, can you 
go that far away from behavioral psychology and stiU 
use programming to achieve your objectives? Well, 
we don’t know for certain, but there is no reason to 
suppose we can’t. Actually, the most important aspects 
of programming as an instructional philosophy are not 
irrevocably bound into the strait-jacket of reinforce- 



ment theory. Indeed, if you read up on some of the 
recent experiments on programming, you will find that 
the cornerstones of reinforcement theory are being 
seriously undermined by the research results. It has 
been shown repeatedly that programs can teach just 
as well without requiring an overt, relevant response. 
Several studies indicate that a program can teach just 
as well without the constant confirmation of correct 
responses. 

Apparently, the really important things that make 
programming successful have nothing to do with the 
two assumptions that comprise the very foundation of 
behavioristic reinforcement theory; the necessity for 
eliciting an overt behavioral response, and immediate 
reinforcement. This suggests very strongly that we may 
be able to expand our horizons beyond a philosophy 
that equates human thinking with the behavioral re- 
sponses of rats and pigeons. That we might, for ex- 
ample, consider the application of the most important 
principles of programming within a transactional psy- 
chology or a motivational psychology, or any other, or 
others, we prefer. 

We must StiU insist on clear statements of objec- 
tives, but perhaps they need not conform to the ri^d 
requirements of a behavioral definition. The clinician’s 
objectives, for example, can rarely be stated in terms 
of specific desired behaviors. We must still insist on 
careful definition of the status quo of the student at 
the beginning of the program, but this might be based 
on diagnostic procedures which use the judgment, and 
the often largely emphatic perceptive skills, of the ex- 
perienced clinician, rather than on a cold inventory 
of isolated segments of behavior. We must stiU apply 
the principle of proceeding in the smaUest necessary 
increments as shown by actual tests. We can stiU 
apply the principle of ruthless elimination of the irrele- 
vant, and so on. But, the appUcation of these principles 
is not limited to an operant-conditioning frame of ref- 
erence; they could be used in a maimer consistent with 
even an existential psychology such as Kierkegaard’s. 
(He was, after all, a great admirer of the Socratic di- 
alogue.) 

What would such a non-behavioral program look 
like? I suspect it would look very much Uke an ordi- 
nary book, but it would be markedly different. It 
would be written with meticulous attention to the learn- 
ing process and would exemplify every principle of 
programming except that it would not necessarily re- 
quire an overt response and constant reinforcement. 
There will be those who will say that it could not 
therefore be caUed a program, but I don’t see why 
not, nor do I much care what it is caUed anyway. 

The behaviorists deserve our congratulations and 
our humble and heartfelt gratitude for their monu- 
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mental contribution in bringing programming into be- 
ing. But there are those of us who find a strictly be- 
havioral view of man unsatisfying and incomplete. It 
would cer tainly be foolish to reject what the behavior- 
ists have taught us, but it would be equally foolish 
to restrict ourselves within the behaviorisfs view of 
the human mind. We cannot claim to think creatively 
about instructional processes until we free ourselves 
to consider the many alternatives offered by the other 
schools of psychology. A word of caution, however. 
This kind of freedom should not tempt us into a state 
of intellectual anarchy. The recent research develop- 
ments do not tell us that we are now liberated from 
the rigorous discipline that science insists on in our 
theormng, our experiments, and our practical tech- 



nology. All these developments tell us is that we need 
not suppose that behavioral psychology has irrevocably 
defined the parameters and the methodology of learn- 
ing science. 

Where we go from here is pure conjecture. It was 
not my purpose in the preceding to prove that either 
rational thinkin g or so-called creative thinking can be 
programmed. I have not yet been eminently success- 
ful at the fixst, and have yet to try the second. Nor 
can I point the direction that programming will or 
should take after leaving its birthplace in the cages of 
the animal laboratory. It would even be foolish to 
assert positively that the door of the pigeon loft is now 
open, but if you look closely, I think you will see that 
it is not locked. 



RESEARCH IN PROGRAMMED INSTRUCTION: AN OVERVIEW 

Leo S. Goldstein 

Leo Goldstein summarizes some of the published research on three programming 
variables: the program, the presentation mode and the learner. The reported studies 
of program variables show inconclusive differences from variation of step size, knowl- 
edge of results or response form. The studies of presentation mode favor the textbook 
form, but only in terms of efficiency, not effectiveness. Of the many learner variables, 
IQ appears to be a good predictor of learning with a program. Students 
overt novel responses appear to learn better than students told to merely think 
their answer. 



Although it is still a relatively young field, a good 
number of articles relating to programmed instruction 
have appeared in psychological, educational, and in-r 
dustrial journals over the past few years. In a recent 
report for the Ford Foundation, Wilbur Schramm esti- 
mates the number of articles on programmed instruc- 
tion appearing in print to be approximately 100. I 
feel that this is somewhat of an underestimate. It would 
be impossible, within the confines of this paper, to dis- 
cuss or even mention all the studies that have been 
completed. 

Therefore, I will try to give you my own “biased” 
view of the field. I will delineate some of the more im- 
portant variables and relate to them some of the re- 
search that has been done. The appendix was de- 
signed to aid you in taking this journey with me through 
the land of programmed instruction. It lists, for ready 
reference, some of the landmarks along the way. 

According to an ancient Roman, “all Gaul is divided 
into three parts.” Programmed instruction also may 
be thought of as a tripartite entity. One major division 
is, of course, the program itself. A second, is the 
method or mode used to administer or present the pro- 
gram. The trinity is completed by the subject or learner 
to whom the program is administered. Each of these 
major divisions may, in turn, be broken down into 
smaller regions. 

Within the program we find, first, the subject matter. 
I would contend that any subject which can be taught 
can be programmed, if the subject matter can be 
clearly defined. This is not to say that programming 
is necessarily the best medium to use to teach any 
particular subject matter. Some information is prob- 
ably better taught via instructional methods other than 
progr amm ed instruction. It is worth noting that a great 
variety of subject matters such as mathematics, the 
sciences, foreign languages, computer programming, 
and even Japanese flower arrangement have been pro- 
grammed. 

The length or number of frames of individual pro- 
grams also has a very wide range. Some programs are 
designed to teach only one or two concepts and, there- 
fore, are extremely short consisting of no more than 



a few dozen frames. Others are intended to teach an 
entire course and they may consist of as many as 
10,000 frames. 

Some researchers contend that certain effects ob- 
served with the use of short programs, such as findings 
of no significant differences between groups using a 
scrambled sequence version of a program and those 
using an unscrambled version, would not appear with 
long programs. This is probably true, but as far as I 
know, a definitive study of this question has not yet 
been made. 

Each programmer has his own style of writing. Many 
use what we call a constructed response type of frame 
in writing a linear program. The material is presented 
in small bits or steps in a logical sequence in which 
each succeeding frame is in some way related to the 
frame, or frames which have gone before; within each 
frame, the learner is required to make a constructed 
response to the stimulus material. After the learner’s 
response has been made, the correct answer is revealed 
to him. This is the so-called Skinnerian style of pro- 
gramming. 

Norman Crowder has developed a style which we 
refer to as “intrinsic” programming. In this kind of 
program, the learner is presented with as many as three 
or four paragraphs of material to read within one 
frame. He then is required to make a response by 
choosing the correct answer from a given multiple 
choice. If the learner chooses correctly, he is sent on 
to the next frame. If he selects an incorrect response, 
he is told that he is wrong and why he is wrong. He 
is given some additional information and is then sent 
back to the original frame to make another choice. 
This kind of programming, when used with the text- 
book format, is the familiar “scrambled text.” Of 
course, these are not the only styles of program- 
ming that have been used. Many programmers prefer 
to be eclectic and will use constructed responses for 
some frames, multiple choice in others; branching tech- 
niques, conversational chaining devices, and any other 
technique which will help them write a good program. 

Of course, now you’d like to know which style of 
progr ammin g is most effective. Truthfully, I don’t 
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know, and I doubt that anyone does. Dr. Susan M. 
Markle at U.C.L.A. is presently doing a comparative 
study of Skinnerian versus Crowderian progr ammin g, 

It’s my feeling that the superiority of any one pro- 
gramming style would be quite dependent on the sub- 
ject matter being taught, the mode of presentation 
used, and certain characteristics of the learner. 

Another facet within the larger division of the pro- 
gram is the step size. Step size is extremely difficult 
to define quantitatively or qualitatively. Let me give 
an example of what is meant by step size. Let us say 
that I wish to teach concept “Q”. I write a program 
which teaches concept “Q” in 50 frames; a second 
programmer writes a program which teaches concept 
“Q” in 25 frames; a third program consisting of per- 
haps 10 or 15 frames designed to teach concept “Q” 
is written by yet another programmer. We would then 
say that the program of 50 frames is a small step pro- 
gram, that of 25 frames is a medium step program, and 
that of 10 or 15 frames is a large step program. The 
learner using the first program proceeds by smaller 
steps than those using either of the other two programs. 
In fact, he needs 50 small steps to learn concept “Q” 
while someone using either of the other programs takes 
fewer and larger steps to cover the same material. Step 
size, then, is a relative measure. Smith and Moore 
(1961) of Bucknell University report a study they did 
with a spelling program. They used a small step, a 
medium size step, and a large step version of the pro- 
gram. Their subjects were the 5th grade public school 
students. The results of the study indicate no signifi- 
cant differences in achievement due to variation in step 
size. Of course, those subjects who used the larger 
step version completed the program more quickly than 
those who used the small step version. It should be 
noted that in this study differences in step size amount- 
ed to only two or three frames per word taught. 

Another aspect of the program to be considered is 
knowledge of results. After the student has read a 
stimulus frame, he is usually required to make a re- 
sponse. After responding, the student has the correct 
answer revealed to him or he is told that the answer 
he has selected is wrong; in some way he is given an 
indication of the correctness or incorrectness of his re- 
sponse. This particular feature of programming is some- 
times referred to as reinforcement or confirmation but 
this terminology is loose in that a response can be 
confirmed only if it is the correct response. An in- 
correct response cannot be confirmed. Revealing the 
correct response to the learner after he has responded 
may be thought of as reinforcing only if this makes it 
more probable that the learner will repeat the response 
at some later time when he is faced with the proper 
stimulus material. A number of studies have been 



done in which knowledge of results was a major vari- 
able. I’d like to mention two of them. The first, using 
Lewis Eigen’s (1960) (1962) Sets, Relations, and 
Functions program was done in the Denver, Colorado 
school system and was then repeated in the Wauwatosa, 
Wisconsin school system. Here, one group was given 
100% knowledge of results. That is, the correct answer 
to each frame was revealed after a response had been 
made. Another group received a random % knowledge 
of results. A third group had a random V 3 of the 
answers revealed. The fourth group received no knowl- 
edge of results at all; they made their responses but 
were not informed directly whether the response was 
correct or not. The results indicated no significant 
differences in achievement due to variation in knowl- 
edge of results. The best predictor of achievement was 
the subject’s IQ. Brighter students learned more than 
duller ones. 

The second study concerned with knowledge of re- 
sults as the major variable was done by Moore and 
Smith (1962) of Bucknell University. Eleven hundred 
and fifty-two (1152) frames of the HoHand-Skinner 
constructed response psychology program were adapted 
to a multiple choice response format. Two hundred 
and fifty (250) college students enrolled in a one- 
semester course in psychology were randomly assigned 
to ten groups. Five of the groups worked with the 
multiple-choice format of the program, and five worked 
with the constructed response format. Eight of the 
groups, four multiple-choice and four constructed re- 
sponse, received instruction via machine presentation. 
The type of knowledge of results varied for each 
group. One multiple-choice and one constructed re- 
sponse group used the program as a programmed text- 
book. 

Four varieties of knowledge of results were used: 
(1) no knowledge of results; (2) immediate knowledge 
of the correct response where the correct answer was 
exposed after the learner had made his response; (3) 
immediate knowledge that the given response was cor- 
rect; a light flashed when the correct response was 
made; (4) immediate knowledge of results plus extrinsic 
reward; a penny for each correct response made was 
given the learner at the end of each instructional 
period. 

Subjects using programmed texts received immediate 
knowledge of the correct response by exposing the 
answer after they had responded. 

The results indicated no significant difference in 
achievement due to variation in type of knowledge of 
results or mode of response. The ^dings are consist- 
ent with those of the Eigen study previously mentioned. 

The second major division which I have delineated is 
the method of administering the program. This may 
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be done via a teaching machine^ programmed textbook, 
index cards, fiim strip, tape recorder or any audio 
and/or visual technique that has been developed. 

Usually, during the creation of a program, the pro- 
grammer writes his frames on index cards. As units 
are completed, the programmer will test his frames on 
individual students in order to get some feedback; some 
immediate information about the effectiveness of what 
he has programmed. After the program has been 
completed, tested, revised, and put into a more durable 
form, it then may be presented or administered by a 
teaching machine. 

Now the teaching machine, of course, is only the 
container for the program. Machines range quite wide- 
ly in size, complexity, and expense. Some of the sim- 
pler and less expensive models are designed to hold a 
program which is usually of the linear style. The 
learner reads the stimulus frame, makes his response 
on an answer tape or within the stimulus frame itself, 
advances the program so that the correct answer is 
revealed, and then compares his response with the 
correct one. On some machines, this advancing of the 
program may be done by turning a knob or pressing a 
button. Machine presentation of Crowderian style pro- 
grams is perhaps a little more efficient than a scrambled 
textbook; the learner using a machine in responding 
presses a button which automatcially presents the next 
proper stimulus frame. This saves the learner time in 
lea^g through the text to find his proper place. Some 
teaching machines have been hooked up to electronic 
computers which tally responses, compute error rates, 
and route the learner through the program. 

Another mode of presentation of programs is, of 
course, the programmed textbook. In addition to the 
already mentioned scrambled textbook used for pres- 
entation of Crowder-style programs, there are two 
basic types of texts for presenting linear programs. 
These are the horizontal and vertical format pro- 
grammed textbooks. The horizontal format uses several 
levels of frames on each page. The learner reads 
across the book, so to speak, on one level, from the 
front to the back, answering each stimulus frame in 
turn. When he gets to the back, he turns the book to 
the front again and starts responding to the frames on 
the next lower level. The correct answer may appear 
on the same page as the stimulus frame or on the next 
page with the following stimulus frame. Wifli the ver- 
tical text format, a number of frames again appear on 
each page but now the learner reads down the page. 
A mask or slider is used to conceal subsequent frames 
and correct answers. A number of studies have been 
reported in which teaching machines have been com- 
pared with programmed texts. Dr. Lassar Gotkin and I 
(1962) have reviewed eight such studies. The general 
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finding is one of no significant difference in achieve- 
ment between these two modes of presentation. Al- 
though no differences in effectiveness have been re- 
ported, in four of the five studies in which time taken 
to complete the program was recorded, a time saving 
of from 10 to 40 per cent was noted in favor of the 
programmed textbook format. It is of interest to note 
that no significant differences were observed, althou^ 
these eight studies varied greatly in subject matter 
and the educational level of the learners. 

Of the three divisions which I have stipulated, the 
most complex, of course, is that of the learner. Here, a 
multitude of factors may be studied; to name just a 
few — age, sex, IQ, educational level, reading ability, 
socio-economic status, ethnic and religious background. 
Some work is being done in programming for handi- 
capped persons. Ted Beckmeyer has studied the use of 
programmed instructional methods with the hard of 
hearing. He has reported that those of his subjects 
who had a reading grade level of at least 3.5 were suc- 
cessful with the program used. Other investigations 
have explored the use of programmed instruction with 
retarded readers and with individuals of low mental 
ability. Dr. John M. McKee is directing a school for 
youthful offenders in an Alabama correctional institu- 
tion. Here, programmed materials constitute the pri- 
mary mode of instruction. AU these special cases are 
a real challenge for programmers and the program- 
ming art. 

In the previously mentioned studies that were done 
in the Denver and Wauwatosa school systems, one 
finding was the importance of intelligence as a pre- 
dictor of success with programmed instructional ma- 
terials. It was found that individuals of high IQ level 
performed significantly better than those with lower 
intelligence. By “performed” I am referring to achieve- 
ment on a criterion test. 

Also related to the learner is his attitude toward pro- 
grammed instruction and his motivation while going 
through the program. A good program is self-motivat- 
ing; it captures the learner’s interest almost immedi- 
ately and carries him along. A number of studies have 
reported on the attitude of learners toward programmed 
instruction. The results are mixed. Some learners 
think programmed instruction is the greatest thing since 
high button shoes; others think it isn’t worth a tinker’s 
dam. In considering these attitudes we have to look at 
them in relation to the kind of subject matter that is 
being taught, the mode of presentation used, and the 
kinds of individuals who are expressing these attitudes. 

I have mentioned several times that the learner going 
through a program reads a stimulus frame and then 
makes a response. The necessity of subjects’ making 
a response is another factor that has been investigated 
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often. These are the so-called overt/covert, or re- 
sponse mode studies. An overt response may be de- 
fined as one in which the learner actually makes a 
motor response and produces some sort of written rec- 
ord which may then be compared with the correct an- 
swer. The learner who responds covertly merely 
“thinks” of what the correct answer might be. Both 
of these modes of response have wide variations. What 
may be an overt response in one investigation may be 
called a covert response in another. In one study the 
overt response may consist of actually writing out a 
word or drawing a figure, or it may be pressing a but- 
ton underneath the right answer. The covert response, 
may be merely “thinking” about what the right an- 
swer is or “thinking” about which button would be 
pressed in order to make the right response. Here, too, 
there is variation in the use and definition of terms. 

I would like to call to your attention a study of 
overt/covert response modes with two kinds of learning 
tasks which were recently completed by AUana Cum- 
mings (1962) of the Center for Programmed Instruc- 
tion. The program used was one which taught the di- 
agnosis of myocardial infarction using electrocardio- 
graphic representation of heart damage. The program 
required the learner to give verbal, that is, written 
responses, and also to draw picutres of electrocardio- 
graph recordings. It was hypothesized that those 
learners who made an overt response to the pictorial 
material, that is, those who actually drew a represen- 
tation of an electrocardiograph recording would per- 
form significantly better on the criterion test than those 
who gave a covert response, that is, those who merely 
thought about what the correct answer should be. The 
results indicate that this hypothesis is tenable since the 
overt responders did perform significantly better than 
the covert responders on this rather difficult, previ- 
ously unknown material. The overt responders also 
performed significantly better than the covert re- 
sponders on the verbal material. This particular find- 
ing was not hypothesized because it seemed to the in- 
vestigators that these verbal responses would be more 
in line with other material already in the learners’ 
repertoires. Responding overtly proved to be more 
effective than responding covertly. 

This, then, has been a very general overview of the 
field of programmed instruction with an indication of 
the kinds of studies that have been done and some of 



the results which have emerged from these investiga- 
tions. I think there are possibly a few generalizations 
that we may make on the basis of these studies. Cer- 
tainly, the findings indicate that programmed instruc- 
tion does teach. There are some people, in fact many 
people who can learn from programmed instruction. 
People new in the field often ask the question, “How 
does programmed instruction stack up against teach- 
ing by the usual methods?” The only answer that 
can be given to this question is, “Which program are 
you stacking against which teacher?” Program ver- 
sus teacher studies are generally not too fruitful since 
the results are not readily generalizable to other pro- 
grams or to other teachers. Arnold Roe, at UCLA, has 
found that students taught by any form of programmed 
instruction such as a machine, text, or programmed 
lecture achieve significantly higher on a criterion test 
than those taught by the standard lecture. However, I 
would not wish to generalize from this finding. I am 
merely calling it to your attention. 

I believe we also may say that comparison of teach- 
ing machines and programmed texts has indicated that 
the text format is as effective as present-day teaching 
machines. This is not to say that this will always 
hold true; quite probably as die development of ma- 
chines advances they will become easier to work with 
and more efficient. For the time being, however, we 
may say that the textbook is as effective in presenting 
programmed instructional materials, and in many 
cases is more efficient than teaching machines. 

Although evidence about the efficacy of overt re- 
sponding is equivocal, it appears, on the basis of the 
Cummings’ study that for certain kinds of subject mat- 
ter which requires a novel (e.g. pictorial or unfamiliar 
verbal) response, actually making a response leads 
to more effective learning than does mere thinking 
about what the response should be. 

In tliis paper, I have necessarily omitted a number 
of other variables involved in programmed instruc- 
tion. The solutions to many of the problems raised are 
still being sought. As time goes on, still more ques- 
tions will be proposed for investigation. If future stud- 
ies can avoid the imprecision of design and definition 
of terms with which some of the pioneer research has 
been replete, we can look forward with hopeful antici- 
pation to reliable solutions for some of these basic 
issues. 
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RESEARCH ACCOMPLISHMENTS AND NEEDS IN PROGRAMMED INSTRUCTION 

Paschal N. Strong, Jr. 

Paschal Strong reports a survey of research studies in programmed instruction, finds 
a few significant implications, many limitations and inadequacies, and indicates the 
areas in which our thinking and methodology has already changed, and the areas in 
which research is badly needed. 



This paper is not an attempt to review exhaustively 
the research literature in programmed instruction. 
Rather, it is an attempt to discuss certain representa- 
tive types of research in order to assess the gains that 
have been made and to point out certain methodological 
and philosophical errors which are prevalent in this 
research field. 

Table I is a summary of 58 studies recently pub- 
lished. They were selected more or less randomly from 
those journals available to the author. i?^Jthou^ they 
do not constitute a complete population, these studies 
should serve as a relatively representative sample. 

Analysis of these studies can be broken down as fol- 
lows: 

1. Subject variables: size, age, sampling procedures, 
education, intelligence, and personality measures. 

2. Program variables: length, method of construc- 
tion, efficacy of program validity, appropriateness of 
program, and subject matter. 

3. Independent variables: branching, program vs. 
“conventional”, machine vs. programmed text, re- 
sponse modes, self-pacing or lock-step, types of pro- 
grams, feedback and reinforcement, step size, and error 
rate. 

4. Dependent variables: pre and post-test compari- 
sons, post-test scores, attitude surveys, time to com- 
pletion, time to complete post-test, retention, and 
scores on various kinds of post-test items. 

SUBJECT VARIABLES: Analysis of the 58 studies 
reveals the following information: 

1. Twenty-one studies used college students while 
nine studies used high school students. Thus, over half 
the references studies came from a highly select, ver- 
bally skilled population. The remammg 28 studies used 
grade school students, except for three which used 
highly skilled, well educated computer technicians and 
14 studies, reviewed by Stolurow, on preliminary 
work with mental retardates. Since these 14 studies 
are not of the parametric type to yield the critical 
data, the final tally indicates that the majority of stud- 
ies represent a small although important segment of 
the population. The implications of this skewness of 
the sample in terms of age, motivation and intelli- 
gence will be discussed later. 

2. The relationship between IQ and programmed in- 
struction performance is not clear. Reed and Haymen 
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in a treatment-by-levels design found that high IQ 
students did better with programmed instruction while 
low IQ students did better with conventional methods. 
However, since the low IQ groups consisted of problem 
students in English, it is highly probably that what is 
called “conventional” instruction is a type or pro- 
grammed (tutorial) course. Differences in reading abil- 
ity and capacity to work independently probably explain 
the superiority of the high IQ group on the English 
2600 program. In well developed programs such as the 
Bell Telephone Labs programs for computer techni- 
cians there is a consistent reduction in the variance of 
post-test scores indicating the ability or programmed 
instruction to reduce the effect of individual differen- 
ces. Although the data are skimpy, it is probably true 
that a good programmed course lessens the effects of 
individual differences in learning. 

3. With the exception of a few studies measuring IQ, 
reading ability or achievement in a specific area such 
as mathematics, little has been done to clarify the 
interaction of subject variables with program vari- 
ables. Of particular interest would be studies relating 
such personality traits as anxiety, need achievement, 
extraversion-introversion, of aspiration and other fac- 
tors to different types of presentation modes, feedback 
systems, and program logic. The same should also be 
done with a more detailed analysis of intelligence such 
as Thurscone’s PMA test, and the subscales of the 
WAIS. In short, too many have assumed that there is 
one best type of program for a particular subject mat- 
ter regardless of subject variables. 

PROGRAM VARIABLES: The greatest weakness 
of research in programmed instruction lies in the area 
of program variables. Inspection of Table I indi- 
cates that in those studies specifying the number of 
frames used, 13 of 44 had less than 100 frames and 
only seven had more than 1,000 frames. Consequently, 
much of the research is based on less than one hour of 
mstructional material and very few were concerned with 
more than ten hours. 

Besides length, another weakness lies in the quality 
of the programs. Except where commercial, on-the- 
shelf items are used, such as the Skinner-Holland pro- 
gram, the English 2600, and the Temac Algebra pro- 
gram, there is little information available on how 
carefully and by what logic the program was derived. 




Scores reported on the criterion post-test in many of 
the studies indicate the programs failed to teach the 
subject matter to the usual programmed instruction 
standards. It is pertinent to ask, I believe, how much 
can be learned about programming from poor, inad- 
equate programs. In many cases the subject matter 
appears to be chosen by expedience rather than by 
the students’ needs. Thus, the question of motivational 
factors also contaminates the results. 

INDEPENDENT VARIABLES: Keeping the previ- 
ous remarks in mind, what has been learned about 
important factors in programming? 

1. Branching. In a series of studies, Silberman, et 
al, investigated several techniques of branching. Gen- 
erally speaking, they and others (Roe, 1962) found for- 
ward branching superior in terms of post-test scores 
and course time, particularly when combined with the 
student’s felt need to branch. A backward review of 
items did not appear to help. 

2. Programming vs. conventional modes of instruc- 
tion. Although several authors, such as Stolurow, feel 
that this is an inappropriate question, I tend to dis- 
agree. The difficulty lies not in the question posed, but 
the failure to define conventional and programmed in- 
struction. In the former, conventional is treated as iE 
analogous to lecturing, while programmed instruction 
is identified as a linear or intrinsic text. Because of 
these errors of definition, the important measures of 
efficacy have been overlooked. Good teachers have 
been implicitly programming their courses for years, 
although most of them would certainly benefit from 
some of the more structured, tested programming tech- 
niques now in use. On the other hand, a person who 
takes a standard text and merely uses it to write a 
series of linear frames is not programming, regard- 
less of the fact that he is supplying feedback, requir- 
ing participation, and using small steps. 

In spite of these confusions and the poor programs 
used, in many cases the evidence is remarkably clear. 
In a series of well done studies on well constructed 
programs, Hughes and McNamera (1961) showed a 
significant increase on post test score, along with a sig- 
nificant reduction of time. Other studies (Smith, Wendt 
and Rust; Hough, Reed and Hayman; Ripple; Legg; 
Green) indicate either better post-test performances, 
shorter learning time, or both. I believe that when ap- 
propriate measures of reliability are used, we will find 
even more stricking differences. 

3. Machine vs. Text. Surely this particular phase 
of research is among the most futile ever done. Con- 
sideration of this problem as a fruitful area for re- 
search can only arise from a profound misunderstand- 
ing of the purpose of a teaching machine. The primary 
purpose of a teaching machine is not to turn pages. 
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present frames, or to prevent cheating; rather, it is 
a device to guide and shape motor responses. Since 
all the research in this area was done with purely 
verbal material the machine was obviously not serving 
its purpose. Programs such as Sandia Corporation’s 
program for typing chemical equations, are probably 
far superior with a machine since the terminal re- 
sponse involves a great deal of precise motor move- 
ments. Incidentally, in this case the teaching machine 
is a typewriter. In this light, it is not surprising that 
no significant differences have emerged from this 
research. Thus the decision of machine or text must 
be made on the basis of economics alone, when verbal 
material are involved. 

4. Response Modes. Most of the research in this area 
has involved either overt vs. covert responding or mul- 
tiple choice vs. written responses. Generally speak- 
ing, the results indicate that the response modes do not 
differ in terms of final scores on a post-test althougji 
the covert and multiple choice modes take significantly 
less time. This research has been aimed at testing the 
importance of active involvement and practicing re- 
sponses. Again, however, there is a failure to define a 
response. We are not trying to teach writing, button 
pushing or talking to ourselves. All of these are merely 
different motor manifestations of the response to the 
problem or question. 

5. Pacing. The few studies done with this variable 
indicate no superiority for the self pacing group. The 
programs listed on Table One that are relevant to this 
question, however, were less than 100 frames. Conse- 
quently, there does not appear to be much room for 
variability m results. Pacing is probably of more im- 
portance in programs requiring at least several week 
work. 

6. Program Format and Logic. Several questions 
have been asked by researchers in this regard. Among 
these are the following: 

a. Can programs be constructed systematically ac- 
cording to a specific set of plans, techniques and pro- 
cedures rather than intuitively? 

b. Should missed frames be retained and presented 
again? 

c. What is the best order for items? 

Although the data are sketchy, it does appear that a 
program can be written more or less routinely by the 
use of the Ruleg principle with as good results as 
those written by the experienced intuitive programmer 
and v/ith less effo^ ! Performance is better if missed 
frames are repeated until correct. Finally, work done 
by Gagne and Brown (1961) suggests that a program 
which guides the student to discover and state princi- 
ples is better than a linear program which gives the 
principle to the student and asks him to learn it. 
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7. Feedback and Reinforcement. The- several studies 
done with this variable all suggest that the use of im- 
mediate feedback or reinforcement (the difference 
between the two terms is difficult to specify) apparently 
confers little benefit to learning. By its very nature, 
however, a good program is probably unable to pre- 
sent a well developed series of frames without rein- 
forcement. Knowledge, properly presented, is prob- 
ably intiinsicly “ri^t” to the subject and when 
knowledge is completely grasped, the subject needs no 
information as to rightness and wrongness. As far as 
the term “reinforcement” is concerned, I believe we 
make a mistake it we discount the reinforcing value of 
learning or attaining knowledge as opposed to being 
reinforced in order to make learning occur. 

8. Step Size and Error Rate. The few studies in 
Table One concerned with step size indicate that a good 
program can be considerably shortened in terms of 
number of frames, which may make step size larger, 
without significantly affecting the scores on the post-test. 
This findin g suggests that most commercially avail- 
able programs, particularly for brighter students, are 
too long, a statement with which most of us who must 
review programs would heartily concur. This occurs 
in spite of a significant increase in program error rate. 

In summarizing the research referenced in Table One, 
it becomes clear that many of the defining phrases used 
to introduce the lay individual to programming have 
little to support them. The following list represents 
some of those phrases or criteria defining a program 
which, in the light of present research, have little ex- 
perimental validity: 1. Small steps, 2. Low error rate, 
3. Immediate reinforcement, 4. Knowledge of results, 
5. Self pacing, 6. Active responding or participation. 
Caution must be exercised in throwing these out, how- 
ever, as we move into programming with different 
subjects and different subject matters. Many of the 
failures to find significant results may well result from 
the limited, highly select subjects used, the extremely 
short and often poor programs, and the inadequate cri- 
terion tests. 

This last point leads us into a discussion of the de- 
pendent variables used, usually a post-test score. 
Since little information is given concerning the con- 
struction of the post-tests, criticism of them is difficult. 
However, it is, or should be, well known that just as 
much time and effort must go into constructing a 
criterion test as goes into defining the program’s ob- 
jectives. Indeed, the criterion test is merely a restate- 
ment of the set of detailed objectives and should com- 
pletely represent that population of facts or knowledge. 
The trial objectives-program-criterion test should be 
essentially identical, with nothing present in one that is 
not present in all. As one member of the triad is re- 



vised, all must be checked to see if this congruency is 
maintained. 

Another major variable used is time, and one of the 
great promises of programmed instruction is that it 
wiU lead to a true science of educational efficiency, 
a highly desirable goal in view of our already over- 
laden educational institutions. Perhaps the addition of 
physiological measures will allow us to expand this 
field by use of the concept of work and physiological 
cost. 

A brief point must be made about retention. Most of 
the research suggests that programmed instruction 
maintains its superiority over time but is not superior 
in terms of percentage retention. This should not sur- 
prise us since the variables effecting retention are 
primarily a function of events occurring after learning is 
over, or events that occurred long before the learn- 
ing took place (retroactive and proactive inhibition re- 
spectively). While a good program may instill knowl- 
edge in such a fashion as to reduce interference of both 
varieties, as yet this goal has not been investigated or 
built into any program known to this author. 

One last critical point must be raised, and this point 
refers to the apparent lack of consideration of the work 
done in the last 30 years on human learning. There is 
almost no citing of the literature concerned with knowl- 
edge of results, meaningfulness, retroactive and pro- 
active inhibition, ego involvement or repetition, to 
mention just a few. Might we suggest a little more 
thoroughness in looking at previous work? An effort 
in this direction should serve to yield better hypotheses, 
save costly repetition of effort, and allow a fusion of 
the two fields rather than a schism. 

A brief statement should be made about future re- 
search in programmed instruction. While it is beyond 
the scope of this paper to discuss this in detail, certain 
broad areas should be indicated. 

1. Greater effort should be expended in relating in- 
tellective, emotional and motivational variables to 
various types of program structures. Of particular in- 
terest should be the intellective factors concerned with 
various sensory modes, and personality variables such 
as anxiety, need achievement, and extraversion-intro- 
version. It will be found, I predict, that these vari- 
ables will significantly interact with such program 
variables as logic (inductive-deductive), step size, 
error rate, feedback, and structure. The ultimate 
goal might be different programs for different per- 
sonalities and intellects. 

2. Since all good programs should lead to almost per- 
fect results on the criterion post-test, the important 
dependent variable will be time and retention as op- 
posed to immediate post-test scores. The inclusion of 
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other measures, particularly physiological measures, 
should be considered. 

3. The most important effort should now be on pro- 
gram format and logic. I am sure all of us feel that 
the present linear, infantile, tippy-toe program is not 
the final answer. Education’s final goal has never been 
solely the attainment of specific knowledge. Rather, 
it has been to give the tools for attaming knowledge 
and arriving at new knowledge. Thus the whole litera- 
ture on transfer of t-aining and creative thought or 
problem solving n.usw be incorporated into the pro- 
gramming field. .1 we do not tie into these other 
streams of research knowledge and ideas, we wiU 
simply end up as a different type of textbook writer 
rather than as contributors to a true science of edu- 
cation. If programs five years from now resemble 
present programs, we will have failed. 

A TABULAR SUMMARY OF RESEARCH 
IN PROGRAMMING 

CODE: E — Experimenter 

D — Dependent Variables 
N — Subjects 

IV — Independent Variables 
C — Comments and Results 

E: Coulson, J. E., Estavan, P. P., Melaragno, R. J., 

and Silberman, H. F. /. Appl. Psychol. 1962, 46, 
389-392. 

N: 30 High school. Frames 233-345 on logic. 

IV: Branching vs. fixed sequence. 

D: Post test score, 51 multiple-choice items and 44 
fill-in items. Time to complete program. 

C: Branching superior in time and score on post test. 

Good criterion for branching errors and self eval- 
uation. 

E: Holland, J. G., and Porter, P. /. Exp. Anal, of 
Behavior 1961, 4, 305-307 
N: 14 college. Frames 2022 Holland & Skinner pro- 

gram on behavior. 

IV : Repeating vs. nonrepeating of missed frames. 

D: 196 item post test immediate and six months re- 

tention. Time. 

C: Review of all items missed superior at all diffi- 

culty levels and/or retention test. 

Time for review adds little to total length of pro- 
gram. 

E: Naumann, T. A., /. Prog. Inst. 1962, 1, 9-18 
N: 44 College. Frames 631. First 16 sets of Holland, 
Skinner 
IV: None. 

D: Time. 

Post test (25 items) plus attitude survey. 

C: Post test poorly described. 
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E: Rothkook, E. Z., /. Prog. Inst. 1962, 1, 19-28 
N: N=96. Frames 60, color coding of resistors. 

IV : Control (just review code) 

Mnemonic 

Mnemonic & Linear Prog. 

D: % correct on post test. 

% correct on 48 hour retention. 

% correct on 30-100 day retention. 

C: % correct all groups =95% thus little room for 

variation. 

Meaningful procedure not used. 

E: Goldstein, L. S., & Gotkin, L. G., /. Prog. Inst., 

1962, 1, 29-36 

Reviews of articles comparmg teaching machines 
vs. programmed texts. 

N: 74 high school, 186 college, 63 technician, 60 col- 

lege, 60 college, 77 8th grade, 87 4th grade, 118 
4th grade. Frames 707 sets & relations, 192 prob- 
ability, 3500 basic electricity, 90 binary, 90 binary 
& psychol., 65 numbers, 3100 spelling. 

IV : Machine vs. textbook 

D: Various post test scores and time to complete pro- 
gram. 

C: No significant difference between programmed 

texts and machines in post test scores. 

Text consistently take less time than machines. 

E: Hughes, J. L. & McNamara, W. J., J. Appl. Psy- 
chol, 1961,45, 225-231. 

N: Maintenance trainees for computer. Frames 719 
on computer flow, logic, etc. 

IV : Lecture vs. P. I. 

D: Post test score. 

Retention test. 

Tune. 

Attitude questionnaire. 

C: P. I. clearly superior on both post test score and in 

time needed for program. 

E: Hughes, J. L., J. Prog. Inst., 1962, 1, 43 
N: 129 technicians. Frames 719 on computer training. 

rV: Reduction of classroom time and method of writ- 
ing answers. 

Overt-covert responding. 

D: 88 item multiple-choice. 

Attitude questionnaire. 

Time. 

C: No significant difference on post test. 

Time reduced 47% over conventional instruction. 

E: Alter, Millicent & Silverman, R. E., /. Prog. Inst., 

1962, /, 55 

N: 60 college. Frames 87 basic electricity. 
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IV: Response mode: 

a. Writing and saying aloud. 

b. Writing +2. 

c. Speaking control. 

d. Thinking groups. 

D: Post test, 22 multiple-choice questions. 

C: No significant difference between response modes. 

E: Alter, Millicent & Silverman, R. E., /. Prog. Inst., 

1962, 1, 55 

N: 60 college. Frames 87 basic electricity. 

IV : Response mode and self pacing vs. forced pacing. 
D: Post test, 22 multiple-choice questions. 

C: Reading better than writing. 

E: Alter, Millicent & Silverman, R. E., /. Prog. Inst., 
1962, 7, 55 

N: 60 college. Frames 90 on binary numbers. 

IV: Overt vs. covert on a text. (The two above were 
machines) vs. machine. 

D: 21 post test items. 

C: No significant difference. 

E: Shettel & Lindley 

N: Frames 30 phonetic alphabet. 

IV : Overt vs covert. 

C: Covert less time. No significant difference on post 

test score. 

E: Roe, A., /. Ed. Res., 1962, 55 
N: 180 college. Frames 3 probability. 

IV: Types of programs: 

1. Linear. 

2. Forward branching. 

3. Backward branching. 

4. Backward alternate. 

5. Pretest forward. 

6. Random order. 

7. Forward branched text. 

D: Learning time. 

Error score. 

Post test time. 

Post test score. 

C: All branching programs same. 

Random order worst. 

Programmed text not as good as branching. 

E: Smith, Nilt, /. Educ. Res., 1962, 55 417-420 

N: 128 college. Frames 337 scrambled books, statis- 

tics. 

IV: Conventional vs. program. Treatments X levels 
design. 

D: Post test performance. 

Attitude questionnaire. 



C: No difference on score. 

Time saved — 30% . Program constructed by 
author. 

E: Wendt, P. R. & Rust, G. D., /. Ed. Res., 1962, 55, 
430-432 

N: 180 college. Frames not given. On use of library. 

IV: Conventional vs. P.T. 

Performance frames. 

Picture frames. 

D: Post test score. 

Time. 

C: No difference on score. 

Time saved very great but no means given. 

Most important part of study mentioned in pass- 
ing. 

E: Evans, J. L., Glaser, R., & Homme, L. E., /. Ed. 
Res., 1962, 55. 

N: 60 college. Frames 75 items on symbolic logic. 

IV: Initial vs. ruleg program. 

Response mode: 

a. Composed. 

b. Multiple-choice. 

c. Covert. 

Delayed vs. immediate feedback. 

Use of review card as auxiliary memory. 

D: A series of parallel test for immediate post test and 
retention test: 

a. 15 item true-false. 

b. 15 item recall. 

c. 15 deductive proof items. 

d. Attitude questionnaire. 

Time to complete program. 

Time to finish post test. 

C: Covert fastest on program, slowest on immediate 

post test. 

Composed made twice as many errors on program 
as multiple-choice. 

No significant difference on post by treatments. 
Note reminiscence. 

E: Eigen, L. D., J. Ed. Res., 1962, 55, 453-460 
N: 78-8th graders. Frames 65 numbers and numerals. 

IV : Machine vs. vertical text vs. horizontal text. 

D: Scores on immediate and delayed post tests. 

C: No significant differences. 

E: Feldhaus, J. F. & Birt, A., /. Ed. Res., 1962, 55, 
461-466 

N: 330 college. Frames 37 frames on teaching ma- 

chines. 

IV : Response modes. 

Feedback. 

Pacing. 

Method of presentation. 
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D: Scores on 15 post test. 

C: No difference between experimental groups. 

E: Hough, J. B., /. Ed. Res., 1962, 55, 467-571 
N: 41 college. Frames- not given subject matter on 
education. 

IV: Teaching machine vs. lecture. 

Constructed vs. selected response. 

D: Two 50 question T-F tests. 

C: Machine superior to lecture on response and time. 

No difference between response modes. 

E: Krumboltze, J. D. & Bonawitz, B., /. Ed. Res., 
1962, 55, 472-475 

N: 32 college. Frames 162 how to write valid achieve- 
ment tests. 

IV: Isolated vs. contextual confirmation. 

D: Post test is 13 short essay questions. 

C: No significant difference between presentation 

modes. 

E: Reed, J. E. & Hayman, J. W., J. Ed, Res., 1962, 
55 

N: About 500 10th grade students. Frames 2632 
(English 2600) 

IV: Text vs. conventional. 

Treatment X level of achievement. 

D: Time for course. 

Score on post test. 

C: High ability did better with P.I. Low ability better 

with conventional procedure. 

Conventional not spelled out. 

E: Lambert, P., Miller, D. M., & Wiley, D. E., J. Ed. 

Res., 1962, 55, 485-494 

N: 552 high school. Frames 843 on sets, relations 

and functions. 

IV: Mode of response X. I. Q. level. Overt vs. covert. 
D: Post test score. 

C: I.Q. difference significant. 

Response mode insignificant. 

I.Q. X response interaction is significant. 

E: Silberman, H. F., Melaragno, R. J., Coulson, J. E., 

& Estavan, D., J. Ed. Psych., 1961, 52, 166-172 
N: 51 high school students. Frames 61 on logic. 

IV: Fixed sequence vs. branching in a P.I. text. 

D: Performance on 24 free response and 24 multiple- 
choice items. 

C: Text branching superior to fixed sequence. Back- 

ward review not significantly better than fixed 
sequence. 

EXPERIMENT I AND EXPERIMENT II 
N: 36 high school students. Frames 411 multiple- 

choice on logic. 

IV: Branching through easier sequences based on er- 
ror rate and type vs. fixed sequence. Matched 



pairs design so that branching member deter- 
mined sequence for non-brancher. Fixed sequence 
means a program unresponsive to individual’s 
error. 

D: Performance on a 51 multiple-choice post test and 
44 free response items. 

C: No significant difference. 

Note that number of frames varied from 69-248 
but no analysis of scores as a function of number 
of frames. 

E: Coulson, J. E. & Silberman, H. F. 

N: 80 E, 104 controls, probably college. Frames 105 

56 on psychology, not further specified. 

IV: Resf»onse mode constructed vs. multiple-choice. 
Branching vs. non-branching. 

Step size. 

D: Post test=19 constructed response and 17 multi- 
ple-choice. 

Retention test in three weeks. 

C: Small step learned more but took significantly more 

time. 

Multiple choice faster than constructed. 

E: DeUa-Piana, L. T., J. Ed. Res., 1962, 55, 502-507. 

N: 125 college students. Frames 280 intro, to coun- 

seling. 

IV: Constructed response, 
cueing. 

Copying correct response. 

Immediate correction of constructed response by 
Procter. 

D: Time. 

Immediate post test and 17-day retention test. 

C: No difference between response modes. 

E: McDonald, F. J. & Allen, D. W., /. Ed. Res., 

1962, 55, 502-507 

N: 122 high school students. Frames 27 rules of a 

game. 

IV: Five combinations of information, examples, ques- 
tions, correction and explanation of corrections. 

D: 22 item post test. 

C: No significant differences. 

E: Taber, J. I. & Glaser, A. J., /. Ed. Res., 1962, 55, 

508-512 

N: Variable. Frames 74 on color names. 

IV: No experiment in formal sense. Just an attempt 
to try various transfer modes. 

E: Shay, C. B., J. Ed. Psych., 1961, 52, 98-107. 

N: 90 4fli grade students. Frames 103, 150, 199 

Roman numerals. 
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IV: Step size of program and I.Q. relationships. 

3 step level, 3 I.Q. levels, balanced design. 

D: Post test of 31 items with 7 new, transfer items. 
Time. 

C: No significant differences. Note that step size de- 
fined by error rate. 

E: Stolurow, L. M. Programmed Instruction for men- 

tally retarded. Rev. Educ. Res., 1963, 33, 126- 
136. 

Reviews a number of studies in this area. 

N: 14 studies here reviewed. Frames. 

rV: Seven deal with language (reading, spelling). 

Five deal with arithmetic. 

Some comparison with normal subjects. 

D: Many uses of humans as part of machine. 

C: Apparently quite feasible for retardates. Consid- 

ering lack of personnel to work with retardates 
this area should be intensively explored. 

E: Cropper, G. L., AID, 1963, 2, 214 

N: Not given. Frames not given (abstract of res.) 

Archimedes law on closed circuit TV. 
rV: Visual vs. verbal program. 

Active vs. passive. 

Visual and verbal in two orders. 

D: Immediate post test. 

Delayed retention test 2-3 weeks later. 

C: Visual better than verbal on visual tests items. 

Reverse true for verbal test items. Total test score 
same. 

Visual/verbal better order than verbal/visual. 
Active superior to passive. Note: This is on a TV 
program! 

E: Ripple, R. E., AID, 1963, 2, 218-220 
N: 240 college. Frames 134 on P.I. 
rV: Responding vs. reading. 

Conventional lecture. Read program to students. 
P.I. but no answers (no reinforcement). 

D: Post test: 

a. 25 multiple-choice. 

b. 25 recall items. 

Parrallel retention test. 

C: Programming significantly superior to lecture and 

reading. 

Reading superior to lecture. 

Reinforcement not necessary. 

E: Bartz, W. H. & Darby, C. L., AID, 1963, 2, 207 
N: 34 college. Frames TEMAC: Algebra II. 

IV: Supervised P. I. 

Unsupervised P.I. 

Traditional lecture. 

D: Purdue placement test. 

Algebra achievement test. 



C: No significant difference btween traditional and 

supervised P.I. 

Unsupervised was inferior. 

E: Moore, J. E. & Smith, W., AID, 1962, 2, 188-189 
N: 96 5th graders. Frames 1108, 838, 596, on spell- 
ing. 

rV: Size of steps: small, moderate, large. 

D: Post test consisted of spelling 50 words. 

C: Large step equally effective as small step program. 

E: Legg, O., AID, 1962, 2, 135 
N: N=480 high school. Frames approx. 500 on agri- 
culture finance. 

rV : P.I. vs. conventional lecture. 

Blackout vs. erasure feedback. 

D: Post test and 2 months retention test. 

C: Both learned, lecture significantly better. 

P.I. 5 hours, conventional 12 hours. 

Erasure better than blackout. 

E: Green, E. J. & Sykes, C. A., AID, 1962, 2, 124-125 
N: 23 medical students. Frames 154 on autonomic 

nervous system. 

rV: P.I. review vs. reading assignments. 

D: 62 item post test by people who did not make pro- 

gram. 

Time. 

C: P.I. superior to reading on both relevant and trans- 

fer items. 

Less time also, so that efficiency of P.I. signifi- 
cantly better. 

E: Asher, J. J., AID, 1961 
N: 80 college. Frames 92 Spanish vocabulary. 

rV: Visually (picture) — aurally vs. aurally — visually. 
D: Score on vocabulary post test. 

C: Visually-aurally superior performance. 

E: Kaes, W. & Zeaman, D., J. Exp. Psych., 1960, 12- 

17 

N: 435 college. Frames 30 psychology definitions. 

IV: Positive and negative knowledge of results. 

D: Number of errors second and third time through 
program. 

C: The more alternatives, the poorer the learning. 

Not really a test of negative KOR. 

E: Cagne, R. M., & Brown, L. T., J. Exp. Psychol., 

1961, 313-321 

N: 33 9th and 10th graders. Frames 89 items + 30 
number series. 
rV: Ruleg program. 

Discovery. 

Guided discovery. 

D: Post test = 4 new problems on series. 

C: Guided discovery best Ruley — worst. 
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THE EFFECT OF WITHHOLDING REINFORCEMENT IN AUTOINSTRUCTIONAL PROGRAMS 

Jainss L. Becker 

James L. Becker’s paper concerns an RCA Educational Services experiment on with- 
holding of reinforcement. The study supports the hypothesis that a person using an 
auto-instructional program in which the steps are kept small and well prompted will 
learn without the usual confirmation. Two groups of students were given identical 
programs except that the experimental group received no confirmation of correct 
responses. There was no significant difference between the acquired knowledge of 
the two groups. 



“I learned a lot,” Ricky commented. “That pro- 
gram stuff is very interesting.” 

The interviewer scanned the post-test; indeed Riciqr 
had acquired a considerable amount from the 200 
frames on Symbolic Logic. “What did you think about 
getting your answer immediately after each frame?” 
the interviewer pressed. Ricky’s bright red hair drooped 
as he assumed a querying countenance. 

“What do you mean?” 

“The answers here on the answer sheet”, said the 
interviewer calmly, pointing to the indicated portion 
of the programmed text. 

“Oh, that,” Rick said, “I hardly ever looked at that.” 

Not many students were as candid as Rick in dis- 
cussing their attitude toward the reinforcement. There 
was other evidence of students ne^ecting the rein- 
forcement. Some branching existed in the program in 
question; yet many students went through routines in 
the program which they were entitled to skip if they 
had bothered to look at the answers. The evidence 
points to the fact that people do not always choose to 
refer to confirmation even when it is readily available. 

Perhaps the phenomenon can be explained by this 
theory. Solving new and challenging problems is rein- 
forcing in itself. The correctness of the solution is ir- 
relevant. As long as an individual thinks that he has 
found an adequate solution to a problem he is rein- 
forced. The many existing contradictory philosophies, 
ideologies and theories and the dedication of their re- 
spective proponents are a living testimony to this fact. 
When a person doubts the validity of his solution, anx- 
iety is bom and he acts to diminish the anxiety. In 
this circumstance we have a definite need for rein- 
forcement. In this situation reinforcement will con- 
tribute to leamiug. If, however, the anxiety is not 
effectively reduced by confirmation, unhealthy ag- 
gression or withdrawal from the learning situation 
may result. 

One hypothesis that can be derived from this theory 
is that a person will learn equally well without confir- 
mation provided the steps are kept so simple that the 
student has confidence in all his responses. This does 
not mean that complex repertoire should not be con- 
stmcted. It only means that new matter must be ade- 



quately prompted at the beginning and that the prompts 
can be gradually and completely withdrawn. 

Recently RCA was developing some materials for the 
Philadelphia National Bank on Time Credit Loans. 
They agreed to allow RCA to investigate reinforcement 
in their program. A program of 180 frames was de- 
signed which consistently used strong prompting and 
total prompt reduction. This program also fulfilled 
entirely the training requirements of the company. 

The following data will aid in further describiug this 
program: 

1. Average time per frame: 22 seconds. 

2. Correlation of acquirement to intelligence: —0.03. 

3. Correlation of intelligence to program completion 
time: -f-0.17. 

4. Mean pre-test score: 33%. 

5. Mean post-test score: 84%. 

6. Size of sample: 10. 

7. Type of sample: Stratified, paired groups. 

The sample was divided into two groups matched ac- 
cording to intelligence, knowledge of the subject mat- 
ter, experience, age and former education. The control 
group was given the program with reinforcement for 
every frame. The experimental group received no re- 
inforcement. Immediately after the program both 
groups were tested. The difference between the mean 
scores of the two groups was 6.7%. This was not sig- 
nificant. 

Although acquirement in both the experimental and 
the control group was significant at the .01 level, the 
difference between acquirement of the two groups was 
not significant. This finding is compatible with our 
theory that reinforcement is only effective when it 
reduces anxiety; however, this does not yet constitute 
proof or adequate substantiation of our theory. Fur- 
ther research will be required. 

The observation that, aside from overt reinforce- 
ment, there are other principle contributing to learn- 
ing in a program, and that these other principles may 
account for much of the acquirement, is an essential 
point for further research. This is important not only 
from a theoretical point of view, but also from the 
purely practical point of view of enabling us to write 
better programs. 
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PROACTIVE AND RETROACTIVE EFFECTS IN PROGRAMMED LEARNING 

J. E. Gerber, Jr. 

J. E. Gerber is interested in developing a way to teach new responses to old stimuli. 
He devised a type of program that uses the old response as a mediator or bridge 
for the new one. The bridge association is pointed out explicity in the content of key 
frames. His data support the conclusion that the explicit bridging approach may work 
well if the student has a good grasp of the old stimulus — response association. 



Inducing students to abandon obsolete associations 
and replace them promptly and accurately with new 
associations is a problem for instructors in subject 
areas in which new developments occur requently. In 
the military, for example, long-held, well-integrated 
concepts of organization, means, methods, and equip- 
ment frequently must be discarded in favor of new 
ones. 

The literature on the dynamics of learning new re- 
sponses to replace old ones presents a somewhat con- 
fusing picture, as may be seen in a comparison of the 
writings of Guthire (1935), Carr (1960), Amsel (1960), 
and Hilgard (1956). But if there is validity in Skin- 
ner’s postulate, that strength of the association (or 
operant) is measured in terms of increased probability 
of response, then teaching the new response without 
extinguishing the old is not likely to strengthen the 
new association to a magnitude significantly greater 
than the old. Stated another way, solely on the basis 
of strength of association, the learner is no more likely 
to respond with the new response than he is with the 
old. And even if he does respond with the new, the 
mere elicitation does not guarantee, according to Blyth 
(1960), that material presented in auto-instructional 
form is either learned or understood. Now if pro- 
grammed material is presented in such small steps that 
the learner can distinguish the new response from the 
old response to the program frames but yet does not 
learn new associations, a conflict in relative strength of 
the old and the new may be postulated. 

The present experiment was designed to test the effi- 
cacy of a method of programming rapidly changing 
instructional material in such a way that for a given 
stimulus a new response rather than a previously cor- 
rect one is elicited and retained. 

Table 1 depicts the experimental design. The 170 
Ss, male junior officers of the Army Medical Service 
Corps, were pre-tested for knowledge of the phonetic 
alphabet used by the Army (Appendix I). Beside each 
of 25 stimulus letters they were to write the phonetic 
response word if they knew it. Most knew some of 
them. The Ss were similarly tested for knowledge of 
25 military symbols used on maps. On the basis of er- 
ror scores on this old phonetic alphabet, 3 groups were 



formed similar as to N, mean, standard deviation, 
median, mode, and range. In each group the model 
error score was 25, the maximum; the modal frequency 
being 10, 10, and 9 respectively for Groups 1, 2 and 3. 
The remainder of each group had at least some knowl- 
edge of this old alphabet at time of pretest on day 
zero. 

Six days after pretest. Groups 1 and 2 studied auto- 
instruction Program 1 on the same, old alphabet on 
which they had been pre-tested. The program was lin- 
ear, 77 frames including criterion test on all 25 letter- 
word associations. (Sample frames are in Appendix I.) 
Group 3 studied a similar program on military symbols. 
All Ss finished the alphabet program well within the 
30 minutes allowed. Criterion test results are shown in 
Table 1. 

Five days after first programs, two forms of a sec- 
ond program were given on a new phonetic alphabet. 
The new phonetic alphabet and sample frames from 
the alternate forms. Program 2A and Program 2B, are 
given in Appendix II. Each program was linear, 65 
frames including criterion test. To the old stimulus 
letters. A, B, C, for example, the new responses. Acre, 
Bolo, China were required rather than the old re- 
sponses, Alfa, Bravo, and Charlie. In Program 2A, 
the “bridging” version studied by Group 1 frames that 
introduced each new response word identified both the 
old and new words, discussed or bridged similarities 
and differences between them, then required S to 
choose between the old and new. Subsequent frames 
required practice of the letter-new word association. 
In Program 2B, the nonbridging version studied by 
Groups 2 and 3, the old word was neither identified nor 
discussed, but S was required to choose and under- 
score the new word in the presence of the old on the in- 
troductory frames. The 30 minutes allowed for comple- 
tion were more than adequate. Results are given in 
Table 1. 

Five days after second programs, new alphabet, then 
old alphabet criterion tests were readministered as 
retention tests. Group results on learning and re- 
tention tests are given in Table 1. Significance of dif- 
ferences of group means is shown in Table 2. 
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For further analysis, Groups 1, 2, and 3 were sub- 
divided on the basis of error scores on the old alphabet 
pretest given on day zero. Those with error scores of 
0-10 (60% or better correct) were called “high knowl- 
edge”; “low knowledge” was 11-25 errors. Sub-group 
means on learning and retention tests and signifi- 
cance of differences between means are shown in 
Table 2. 



Two conclusions with regard to bridging as used in 
this study are offered: 

1. Bridging apparently facilitates retention of new 
responses by subjects with well-established high knowl- 
edge of old responses to common stimuli. 

2. Bridging apparently lowers retention of new re- 
sponses by subjects who have little or no knowledge of 
the old responses to common stimuli. 
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Table 2 

Significance of differences of means of error scores of the groups and subgroups 
on new alphabet criterion and retention tests 
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Appendix I 

“Old” phonetic alphabet and sample frames 
from Program 1 



ALFA 


JULIETT 


SIERRA 


BRAVO 


KILO 


TANGO 


CHARLIE 


LIMA 


UNIFORM 


DELTA 


MIKE 


VICTOR 


ECHO 


NOVEMBER 


XRAY 


FOXTROT 


OSCAR 


WHISKEY 


GOLF 


PAPA 


YANKEE 


HOTEL 


QUEBEC 


ZULU 


INDIA 


ROMEO 





Program 1 Card 3 



The first letter in the alphabet is A, and the word 
used for A in the phonetic alphabet is ALFA. 
Write ALFA on the solid line. 



Program 1 Card 15 

Now see how well you Vave done. (Don’t go back 
to earlier cards. Try .o remember.) 

Tbe word for A is 

The word for B is 

The word for C is . 



Appendix II 

“New” phonetic alphabet with sample frames from 
the “bridging” (2A) and “non-bridging” 

(2B) programs. 



ACRE 


HILLTOP 


ORANGE 


VIOLET 


BOLO 


INDIGO 


PEPPER 


WISH 


CHINA 


JULY 


QUOTE 


XRAY 


DADDY 


KITE 


RIP 


YELLOW 


EERIE 


LULU 


SIMPLE 


ZOO 


FUR 


MONK 


TOT 




GORGE 


NIKE 


UNDER 





Program 2 A Card 3 

THE NEW PHONETIC FOR LETTER A IS 
ACRE. ACRE HAS THE SAME NUMBER OF 
LETTERS AS ALFA, THE WORD YOU AL- 
READY KNOW FOR LETTER A. IT IS THE 
WORD, ACRE, THAT YOU ARE TO LEARN. 
NOW, STUDY THE TWO WORDS BELOW: 
UNDERSCORE THE NEW WORD FOR LET- 
TER A. 

ACRE ALFA 



Program 2B Card 3 

THE PHONETIC WORD FOR LETTER A IS 
ACRE. ACRE IS A 4-LETTER WORD BEGIN- 
NING WITH LETTER A. IT IS THE NEW 
WORD, ACRE, THAT YOU ARE TO LEARN. 
NOW, STUDY THE TWO WORDS BELOW: 
UNDERSCORE THE NEW WORD FOR LET- 
TER A. 

ACRE ALFA 



Program 2A Card 7 

AFTER BOLO COMES CHINA, THE WORD 
FOR LETTER C. YOU HAVE LEARNED 
CHARLIE FOR LETTER C IN THE OLD 
ALPHABET, BUT YOU MUST NOT USE 
THAT WORD HERE. CHINA IS THE WORD 
FOR LETTER C. 

UNDERSCORE THE WORD FOR LETTER C. 
CHARLIE CHINA 



Program 2B Card 7 

AFTER BOLO COMES CHINA, THE WORD 
FOR LETTER C. CHINA IS THE WORD FOR 
LETTER C. 

UNDERSCORE THE WORD FOR LETTER C. 
CHARLIE CHINA 



EXPLORATIONS IN STUDENT-CONTROLLED INSTRUCTION 

Robert F. Mager and Cedi Clark 

Robert F. Mager and Cedi Clark describe a number of studies possessing common 
features in which students controlled the learning experience. Learners were encour- 
aged to indicate which topics should be discussed, how deeply they should be 
explained, what equipment or demonstrations were required, the frequency of reviews 
and the length of instructional periods. Instructors respond to questioning but do not 
provide unsolicited information or explanations. Findings showed a decrease in 
training time, variations in content sequencing, an increase in student competence and 
confidence, and achievement comparable to conventional instruction methods. The 
authors recommend that techniques be developed for the determination of the back- 
ground level of students permitting individualized curriculum adjustment which will 
induce an economic progression to the course objectives. 



The developmpnt of programmed instruction is, or 
certainly should be, a dynamic undertaking. While 
we may differ widely and even wildly on what pro- 
gramming will ultimately be like, few would question 
that we have a way to go before we attain the ultimate. 
In short, some changes are indicated. The interesting 
thing about this is that, although most of us will ac- 
knowledge that change is necessary, we are ignoring 
some of the facts that indicate where changes should 
be made. 

In this paper, we want to suggest an area in which 
program design can be improved. We shall try to ac- 
complish our purpose by briefly describing a few stud- 
ies, identifying some common features of these 
studies, and then by suggesting what course of action 
is indicated. As our discussion proceeds you will un- 
doubtedly be struck by the fact that there appears to 
be little, if any, new information. Even the conclusions 
we draw from the data have been known about and 
talked about educators for years. What would be 
new and what we advocate would be the widespread 
application of these well-known facts; and what would 
be novel would be for us to practice what we preach. 

Most of the studies we will describe are those in. 
which the learners were given a good deal of control 
over the learning experience. As a matter of fact, it 
was this very feature in these experiments that led us to 
rediscover the astonishing phenomenon that adult 
learners enter a formal learning situation with previ- 
ously gained knowledge relevant to the learning at hand. 

The first study was undertaken about four years 
ago by Mager (1961) to discover whether a 
learner would sequence instruction in the same way 
that an instructor typically sequences it. If the se- 
quence of information called for by a learner was radi- 
cally different from that prescribed by an instructor, 
it might be inferred that a course sequence genen’ted 
exclusively by the instructor or the expert was less 
meaningful to the student than it might be. To explore 
this problem, one student at a time was given com- 



plete control over a curriculum in electronics. The 
instructor tried to behave like a mechanism that would 
respond to the student’s questions, but which would 
offer no imsolicited information or explanation. The 
learner was encouraged to indicate which subjects he 
wanted to discuss, how deeply he wanted to have them 
explained, the kinds of equipment or demonstrations 
he felt would be useful, and the frequency and nature 
of reviews. The learner also controlled the length of 
each instructional session. Results of this study indi- 
cated that the sequence of information called for by 
the learner was considerably different from that of the 
traditional electronics curriculum. Also, the fact that 
the student had a good bit to say about the procedure 
seemed to exert a strong and favorable influence upon 
his motivation. These results were indeed interesting 
and provocative, and have led to further studies and to 
applied progr ammin g. In the programming depart- 
ment of Varian Associates, for example, at least two 
subjects are now interrogated before any program 
frames are written so that guidance relative to infor- 
mation sequencing can be obtained from members of 
the target population. 

In addition to the results just described, a rather nag- 
ging phenomenon was repeatedly observed. No matter 
how ignorant the learners appeared to be, no matter 
how slowly they appeared to learn, no matter how 
naive they claimed to be, male and female subjects 
each entered the experiment with some relevant knowl- 
edge about electronics. Some of the subjects knew 
more than others, of course, and one or two had devel- 
oped some rather interesting misinformation about the 
subject. Nonetheless, no subject started with zero re- 
levant knowledge. 

The second study (Mager and McCann, 1962) was 
an applied industrial experiment conducted in one of 
the manufacturing divisions of Varian Associates. The 
study involved an engineering course, six months long, 
given newly graduated engineers before they were 
permanently assigned to their positions. Training in- 
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eluded theory, machine and instrument operation, 
manufacturing processes, and company procedures. 
The first six weeks of the course were devoted prim- 
arily to formal lectures; during the second six weeks 
each trainee was rotated throu^ the various depart- 
ments through temporary assignment to the man- 
ager or expert in each of the departments. During the 
last three months of the course the student was assigned 
as an assistant to an experienced engineer. Because 
of the small classes involved (four to eight students), 
each student received what amounted to six months of 
individual instruction. 

Our experiment again called for a good deal of con- 
trol on the part of the student. All classes were can- 
celled and the instructor was asked to “speak only when 
spoken to.” Students were told that they would have 
complete control over what they learned, when they 
learned it, and how they learned it. They were told 
that they could ask for instruction from anyone in the 
division, but that they were not to accept instruction 
they did not want. One important difference between 
this study and the previous one was that this time the 
students were given 24 pages of detailed course objec- 
tives which specified the desired terminal behavior. 
The total effect was that the students had to decide 
what they needed to learn in addition to what they al- 
ready knew in order to reach the objectives. 

Some of the results of this study were: 

a) Tr ainin g tim e was reduced 65 percent. All stu- 
dents were permanently assigned six to eight weeks 
after training began. 

b) The graduates of this program appeared better 
equipped than those of previous cycles, demonstrating 
more confidence and more knowledge about manufac- 
turing processes and microwave theory. The man- 
ager of the division considers these engineers “better 
trained” at the time they assumed their permanent 
responsibility than previously hired engineers. 

c) Considerably less time was devoted to training by 
the instructor, administrators, and technical experts. 

d) Content selected for study varied considerably 
from student to student. 

e) The sequence of information studied varied from 
student to student, but in no case coincided with the 
sequence used previously. 

Though no instructional programs were used in this 
study, it still appeared possible to significantly reduce 
the length of a formal training program, while at the 
same time improving the competence and confidence 
of the student. This result was obtained primarily by 
providing the student with a detailed description of the 
desired terminal behavior, and by allowing him to 
fill in the knowledge gap between what he already knew 



and what was required. Put another way, he was not 
required to relearn what he already knew. Since 
some of thse students had been previously trained as 
electrical engineers, some as industrial engineers, and 
some as mechanical engineers, it was known that they 
entered the course with a considerable amount of rel- 
evant knowledge, which would help account for their 
rapid progress. It was noted, however, that students 
in previous cycles also came in as trained engineers. 

A third study (as yet unpublished) that we would 
like to mention also investigated content sequencing. 
This study used the skill of electrical meter reading 
as the vehicle. During the early phases of the study 
we wanted to determine empirically the meter read- 
ing skill of several kinds of experts. An automated 
meter reading examination was thus administered to a 
group of physicists, a group of engineers, and a group 
of technicians. The final score for each of these groups 
was approximately 80 per cent. When housewives were 
given the same examination without any prior train- 
ing, their average score on the examination was 40 
per cent. Again, although these subjects all claimed 
ignorance in this area of electronics and meter read- 
ing, and although they had no formal training this 
or related areas, their performances were significantly 
better than zero and half as good as those of the 
experts. Unlike the previous study, here is an example 
of a group of students entering a training situation sup- 
posedly with a minimal amount of relevant information; 
obviously, however, these housewives possessed a con- 
siderable amount of relevant knowledge. 

Still another experiment (Mager and Qark, 1963) 
provided results indicating that the expert was no bet- 
ter than the naive subject. The object of this partic- 
ular study was to determine whether a qualitative 
knowledge of result could be used to teach a subject in- 
formation for which there was no right or wrong an- 
swer. Specifically, the object was to determine whether 
subjects could learn to discriminate different smooth- 
nesses of metal surfaces when their responses were 
confirmed by the percentage of expert agreement with 
the response rather than by an indication of correctness 
or incorrectness. To complete the device used in the 
experiment, it was necessary to obtain expert judg- 
ments. The stimulus items were therefore judged by 
thirty-six experts and their average judgment wired into 
the machine. For this experiment ai “expert” was de- 
fined as a person who had final authority over the dis- 
position of the metal materials in question, rather than 
in terms of skill. 

The result of this study that is relevant here is that 
the performance of the experts turned out to be identi- 
cal in every respect to that of the control group receiv- 
ing no training whatsoever. 
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While the previous experiments we described suggest 
that students frequently know more than we give them 
credit for, the results of this smoothness experiment 
suggest that some experts know a good deal less than 
we give them credit for. In other words, the behavior 
of the expert, or should we say instructor, may not be 
as different from that of the layman as he would like 
to have us believe. But while it may be more com- 
fortable to talk about engineers in this way than about 
ourselves, we have similar problems closer to home. 
There have been countless studies, for example, in 
which the ability to predict human behavior was stud- 
ied as a function of training and experience level. The 
ego-deflating results have uniformly demonstrated that 
the predictions of those naive organisms commonly re- 
ferred to as “college sophomores” and “underpaid sec- 
retaries” were essentially as accurate as judgments 
made by highly trained and experienced clinical psy- 
chologists. The expert, in other words, didn’t make 
judgments any better than the novice. We also suspect 
that the judgments of “experts” in other disciplines, 
if subjected to a reality test, might not justify the con- 
fidence or the credibility attributed to their pronounce- 
ments. 

In an experiment conducted at Stanford last spring 
(Allen and McDonald, 1963), subject were required to 
learn the pieces, rules, and strategies of a new game. 
One group learned from a linear program, while each 
member of another group was provided with objectives 
and an instructor that he could turn on and off at will. 
The members of the group which had control of the 
curriculum performed almost as well as those learn- 
ing from the linear program, but they required only 
half the instruction time required by those using the 
program. A peculiar observation made during this 
study was that while the subjects of the experiment were 
bright Stanford students, those in the group which 
had control of the curriculum exhibited no skill at sys- 
tematic information collection. In other words, had 
you watched these subjects you would probably have 
predicted poor terminal performance because of the 
chaotic way in which they asked questions of their in- 
formation source. Nonetheless, this group performed 
almost as well as the program group, in half the time. 

The general conclusion we draw from these studies 
is that adult students are likely to enter a learning sit- 
uation with a significant amount of relevant knowl- 
edge; in other words, they are likely to already know 
something about that which is to be taught. 

Let us examine some possible implications of such a 
conclusion. Is this new information? Certainly not! 
One of the oldest rules of teaching exhorts the teacher 
to “take the student from where he is to where you 
want him to be.” Presumably this means that one 
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should start instruction at the edge of the student’s 
knowledge; to start beyond the edge of his knowledge 
is to construct unnecessary obstacles in the path of his 
learning, and to start within his knowledge is to bore 
him. The incredible thing is that we are well aware 
of the existence of differences in student knowledge 
and experience. At the same time, however, we behave 
as though we wished this difference would go away. 
We are willing to recognize that students enter in- 
struction in various stages of wn-preparation, and we 
apply remedial procedures to the learner to minim ize 
his lack of preparation. But the questions to be raised 
are, why don’t we also recognize that students may 
enter ins truction in various stages of over-preparation, 
and why don’t we apply remedial procedures to the 
curriculum? Certainly we admit the existence of 
“overpreparation” every time we decide to “control” 
an experiment by using a pre-test. Yet, in program- 
ming, the overwhelming tendency is to cause each 
learner to work his way through exactly the same se- 
quence of information; this is true for both linear and 
branching programs, even though the branching pro- 
gram allows students to omit a good many remedial 
discussions. What we seem to have overlooked is the 
possible advantage to be gained by subtracting from his 
curriculum that v/hich the learner aheady knows. There 
is, after all, no reason to believe that the only way to 
obtain further improvements in instructional efficiency 
is through the improvements of programs and teach- 
ing machines! We submit that it is timely to begin 
thinking about curriculum-generating machines! These 
devices would be designed to detect what the student 
already knows, compare this body of knowledge with 
that required by the objectives of the program, and 
then generate a curriculum for the student. The result 
would be a saving in student time equal to the time it 
would take to teach him what he aheady knows rela- 
ting to the instructional objectives, minus the time re- 
quired to detect the state of his knowledge; boredom 
would also be reduced. 

There is another possible implication that can be 
drawn from our conclusion. In the absence of a cur- 
riculum-generating machine, the adult learner himself 
might be a better judge of what he needs to add to his 
current knowledge in order to reach some given set of 
objectives than is a textbook writer, instructor, or pro- 
grammer. Given half a chance and a set of reason- 
able objectives, he will probably generate for himseff 
a curriculum that will lead him to achieve these ob- 
jectives. Interestingly, this implication is not without 
support. There are several studies performed by pro- 
fessors (Duke, 1959; Milton, 1959; Weitman and Gru- 
ber, 1960) who have allowed one college class to attend 
all the lectures while prohibiting another group from 
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any lecture attendance. Even in the absence of care- 
fully specified obejctives, students of the group who 
were not allowed to come to class performed just as 
well as those in attendance. While this is not conclus- 
ive evidence that teachers aren’t necessary, the data 
suggests that we might improve the efficiency of in- 
struction by making better use of the intelligence 
and background which the adult student brings to the 
formal instructional experience. 

Thus, the new frontiers in program design that ap- 
pear to be fruitful are these: 

1. The development of effective methods for detect- 



ing the relevant background level of the incoming stu- 
dent, and for adjusting the curriculum accordingly. 

2. The development of techniques which the learner 
himself can use to efficiently generate a curriculum 
which will lead him as economically as possible to 
the objectives. 

To summarize, while we feel that further improve- 
ment in progr ammin g can be achieved by continued 
research with variables having to do with subject 
matter presentation, we suggest that greater improve- 
ments can be attained by focusing on variables relating 
to subject matter selection. 
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PARTIAL AND CONTINUOUS FEEDBACK IN A LINEAR PROGRAMMED INSTRUCTON PACKAGE 

Walter E. Driskili | 

Walter Driskili investigated the eflFects of partial and continuous knowledge of ' 

results in a linear program. One group of students received 100% feedback/ the ^ 

other group received feedback on an aperiodic 40% of the frames. Prior laboratoiy | 

research suggests that continuous feedback should enhance learning, but aperiodic ] 

feedback should increase retention. Mr. Driskili observed no such differences. Learn- i 

ing and retention scores were almost identical for both groups. . | 



In the literature on programmed instruction there is 
almost unanimous agreement that the provision of im- 
mediate knowledge of results constitutes reinforce- 
ment of a learner’s responses and contributes materi- 
ally to his learning. This imm ediate feedback feature 
is frequently cited as being one of the primary require- 
ments for an effective linear program (Porter, 1957; 
Carr, 1955;MerriU, 1961). Skinner (1954, 1958, 1960), 
in establishing some of the principles of linear pro- 
granuning, stated that “immediate feedback (is used) 
not only to shape (a learner’s) behavior but to 
maintain it in strength.” A lone dissenter in the 
role of feedback is Crosier (1960) who proposes that 
knowledge of results may be desirable, but feedback in 
his system is not provided for its reinforcement effects. 
The primary reason for providing knowledge of results 
is to determine whether the communication of informa- 
tional material has been successful. - - 

Programmed instruction packages being prepared 
today, both commercially and in the military, without 
exception provide for immediate confirmation of 
the correctness of a learner’s responses. Even branch- 
ing type programs contain feedback information that 
may have reinforcing values. 

The importance of reinforcement in facilitating learn- 
ing has been established in many studies (Woodworth, 
1954). More specifically, some of the early experi- 
mental work in programmed instruction dating from 
the time of Pressey (1926) and Little (1935) indicate 
that in a test situation those students receiving feed- 
back concerning the correctness of their answers 
learned more efficiently than students who did not 
receive such feedback. 

There is a comparable body of literature dealing 
with the question of partial and continuous feedback. 
This literature was reviewed by Jenkins and Stanley 
(1950) and Lewis (1960). These reviews suggest that 
response strength is generally greater imder conditions 
of continuous feedback during acquisition but resist- 
ence to extinction is greater when partial reinforcement 
conditions are used. 

Because these general findings pertain to sched- 
ules of reinforcement, a study was undertaken to deter- 
mine the effects of partial and continuous feedback 
in a linear programmed instruction package. 



One hundred ninety-four airmen undergoing basic | 
military training were assigned to two treatment groups. j 

Airmen were not individually assigned but the flints | 

to which they were randomly assigned upon entering | 

training were randomly assigned to the two treat- # 

ment groups. These airmen completed a 52-frame | 

linear program on powers and roots of numbers. This | 

program provides instruction on the derivation of | 

powers and roots of numbers, exponents, and fractional | 
and decimal exponents. | 

The linear program was presented in two ways. One | 

group of 97 airmen comprising the regular reinforce- | 

ment group completed the program as it was originally | 

developed. Each learner responded in each frame and j 

was able to turn to the next page and compare his i 

response with the correct response provided there. He j 

was able, in other words, to confirm the correctness of | 

his response immediately after he had responded. The | 

second group of 97 airmen completed the program | 

which had been altered to provide feedback only on | 

40 per cent of the frames on a periodic basis. The first | 

ten frames remained as they were originally developed | 

with the correct response printed on the page following i 

the frame. The remaining 32 frames were presented | 

so that the correct response was deleted from 70 per- | 

cent of the frames. | 

Three parallel forms tests were devised. Each was | 

a 12-item test which measured specific principles in- j 

eluded in the power and roots program. The first test | 

was used as a pre-test administered prior to training. | 

The second test v/as administered immediately after the j 

trainees had completed the 52-frame program. The j 

third test was used as a measure of retention and was | 

administered seven days after the learner had completed j 

the program. Comparability of groups was determined j 

by use of general aptitude index scores taken from the | 

Airman Qualifying Examination. The groups did not | 

differ significantly in aptitude, the means scores being | 

53.1 for the regular group and 54.6 for the irregular | 

reinforcement group. | 

Scores that each learner made on the pre-test, post- | 

test, and retention test were recorded. Mean scores j 

for each test were computed and t-tests of differences | 

between mean scores on each of the tests were made. j 

The resulting t ratios are shown in Table 1. Here is | 
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seen that no differences between treatment groups oc- 
curred for pre-test, post-test, or retention test. 

In Table 2 the achievement gain for each reinforce- 
ment group is shown. Each group made gains signifi- 
cantly beyond the .01 level. This gain was computed 
by taking the differences between the pre-test and post- 
test scores for each student. 

In Table 3 the extent of retention loss is shown for 
each reinforcement group. As shown, neither loss is 
significant. Likewise, the difference in loss between 
groups is not significant. 

As a result of these findings, the conclusion is that 
for this program continuous feedback did not facilitate 
learning nor did irregular reinforcement facilitate reten- 
tion. These findings are contrary to the general find- 
ings pertaining to the question of partial and continu- 
ous feedback. 



Another conclusion is that the program did an 
efficient job of instructing since significant gain was 
noted for each group and the retention loss after seven 
days was very slight. 

As a result of this study, the question naturally arises 
concerning the requirement for continuous feedback. 
More pertinently, it may raise a question as to what is 
the role of feedback in a linear program. It is suggested 
here that feedback is a reinforcing agent in the linear 
program until the student has learned that he can 
respond correctly by using the cues and prompts that 
are provided in frames. Feedback provided early in a 
program may reinforce correct responding behavior. 
Once this behavior is established, feedback may be 
unimportant in the remainder of the program. The 
theoretical implications for learning theory and for pro- 
gramming instruction of this possibility should be 
explored. 
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Summary Analysis of Test Scores for Each Reinforcement Group 


















Mean 




Test 




Regular 






Irregular 




Difference 


t-ratio 




N 


X 


S 


N 


X 


S 






Aptitude 


97 


53.1 


— 


97 


54.6 


— 


.1 


.193* 


Pre-Test 


97 


3.2 


2.4 


97 


3.3 


2.5 


Post-Test 


97 


8.8 


4.7 


97 


8.1 


4.4 


.7 


1.910* 


Retention Test 


97 


8.0 


2.9 


97 


7.5 


2.9 


.5 


1.20.>* 


* Not significant at the .05 level (two-tailed) 















TABLE 2 

Achievemen} Gain for Each Reinforcement Group 

Pre-test Post-test 

Mean Mean 



Regular 

Irregular 



3.2 

3.3 



8.8 

8.1 



* Gain is significant beyond .01 level 



Gain 

5.6* 

4.8* 



TABLE 3 

Retention Loss for Each Reinforcement Group 

Post-test Retention test 
Mean Mean Loss 

Regular 8.8 8.0 .8* 

Irregular 8.1 7.5 .6* 

♦ Neither loss is significant at the .05 level 

(Note: The difference in loss between groups is 
not significant.) 



AN EVALUATION OF PROGRAMMED INSTRUCTION AND 
CONVENTIONAL CLASSROOM TECHNIQUES 

Maurice A. Larue, Jr. and F. Elaine Donelson 

Maurice A. Larue and Elaine Donelson of the Martin Company performed a com- 
parative experiment on the efficiency of programmed instruction using conventional 
instruction, a programmed text and a teaching machine on a 1 9-hour segment of the 
"Fundamentals of Computers" in a weapon system training course. The experiment 
showed no significant increase in learning using programmed instruction, but it did 
show 25 percent time savings. The experiment forbade the experimental groups from 
studying at home, but permitted the control group to study at home. 



With programmed instruction being heralded as a 
new and efficient tool in the technology of learning, we 
at the Martin Company were interested in determining 
whether or not this technique would have application 
to our weapon system training program. 

An experimental program was designed and initiated 
to determine primarily the efficiency of using this tech- 
nique to teach, and secondarily, to investigate variables 
which may influence learning with programs. The pri- 
mary questions, which are the focus of this paper, were: 
(1) Can a programmed course teach?; (2) If so, to 
what level of proficiency does it teach compared to 
conventional classroom instruction?; (3) Does it take 
more, less, or equal time, as compared to conventional 
time to attain this level of proficiency?; and (4) Are 
the answers to these questions dependent upon the 
presentation technique of the programmed course? 

Additionally, secondary questions included whether 
or not relationships would show up between education, 
verbal and mathematical aptitude, attitude, and gain 
and time. These questions are discussed in a report 
published by Martin Orlando (Larue, Donelson 1963), 

Because our chief training needs are weapon system 
training, a segment of the existing Pershing Weapon 
System New Equipment Training Course was selected 
as the course material for the experiment. This seg- 
ment, “Fundamentals of Computers,” was a nineteen 
hour classroom portion of instruction on the Pershing 
Fire Data Computer and was subdivided into six les- 
sons, The terminal objectives for each of the lessons 
had been predetermined by Pershing Training per- 
sonnel and established in the Plan of Instruction (POI) 
for each lesson. 

This material was programmed using intrinsic or 
branching techniques; the behavioral objectives speci- 
fied in the POI were retained in full. The six lessons 
and the objectives of each were: 

1. Number Systems: The objective of this lesson was 
to make the student aware of various number systems 
such as the binary, tertiary, octal, and decimal. He 
was to learn the terminology peculiar to number sys- 
tems (radix, coefficient, etc.) and how to express a 
number in a particular system mathematically. 



2. Number Systems Conversions: The objective of 
this lesson was to teach the student to convert both 
whole numbers and fractions from one number system 
to another. The number systems involved were deci- 
mal, binary, and octal. 

3. Binary Arithmetic: The objective here was stu- 
dent mastery of addition, subtraction, multiplication, 
and division of decimal, binary, and octal number sys- 
tems, Knowledge of complements in each system was 
also an objective, 

4. Computer Language: Here, the student was to 
become familiar with computer terminology. He was 
to learn what a general purpose digital computer con- 
sisted of and the relationship and function of each of 
its units. 

5. Computer Logic: The objective here was to teach 
the student the general logic associated with a general 
purpose digital computer such as the relationships 
which existed between logical “1” and logical “0” 
inputs and outputs for various gating circuits. 

6. Computer Building Blocks: The objective of this 
lesson was to teach the student the function and use 
of the seven major computer building blocks used in 
the Pershing system. 

A behavioral objectives test was prepared for the 
program. It was of the same multiple-choice format 
used throughout the branching program and was de- 
signed to have at least one question per each prime 
frame in the course. Preparation of the test in this 
manner allowed its use both in the preliminary evalua- 
tions of the programmed course and in the evaluation 
of student performance. 

Subject matter experts and the Pershing training in- 
structors reviewed the programmed material and, after 
their suggested changes were incorporated in the 
courses, endorsed its technical validity. The program 
was then tested empirically with students. For the ini- 
tial test, four subjects, selected to represent the range 
of the target population, took the course presented in 
a “scrambled book.” On the basis of their comments 
and the item statistics provided by the objectives test, 
revisions were made in the program. 
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Six subjects, selected jErom men assigned to take the 
course as part of their training, were used to test the 
revised program. On this test, presentation of the 
material was by machine. Their performance and com- 
ments resulted in addition revisions. For the experi- 
ment itself, the material was presented to the students 
in three ways. The classroom, or control group, re- 
ceived conventional platform lectures. The book ex- 
perimental group used 6'^ x 814^^ spiral bound scram- 
bled books designed so that information on one page 
was not directly related to information on the adjacent 
page. The machine experimental group used the U.S.I. 
Mark II Autotutor. The course content in all three 
cases was the same, and was designed to provide mas- 
tery of the same behavioral objectives. In the case of 
the two experimental groups, the material was identical 
with the exception of information relating to machine 
and book operating procedures. 

A total of 70 subjects was used. AU were adult male 
military personnel who were required to have an EL 
score of 100, shown by actual test scores or ofiScer 
status. Fifteen were available to serve as the conven- 
tional group; thirty for the machine group and 25 for 
the book group. All three groups were tested for their 
knowledge of the course material both before (pre-test) 
and after (post-test) taking the course. The previously 
mentioned behavorial objectives test consisting of 174 
multiple-choice questions was used in both cases. 

The conventional group received 19 hours of regular 
classroom instruction and could study at home as they 
wished. Both programmed instruction groups were 
told to go through the course as fast or as slow as they 
felt was needed to master the material. For these 
groups no instructor interaction was allowed nor was 
any home study permitted. 

Knowledge of the course material before instruction 
was compared to knowledge after instruction to deter- 
mine if the programmed course had taught. For both 
the machine and book groups, achievement scores 
after instruction were significantly higher than before 
(p <.01 ); learning did occur as a result of using the 
programmed material. The classroom group also 
showed a significant increase in knowledge as a result of 
their instruction. 

Post-test scores were analyzed to determine if there 
were any differences in the achievement level attained 
by the three groups of subjects receiving the instruction 
by the conventional, machine, and scrambled book 
methods. The influence of initial differences in knowl- 
edge of the material was controlled by covariance, 
which increased the precision of a sunple analysis of 
variance by 59%. No significant differences occurred 
among the three groups; there is, then, no reason to 
say that one group learned more than another. Table 1 
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presents the means and variability measures of each of 
the groups on the pre- and post-tests. Also shown are 
the post-test means adjusted, by the covariance tech- 
nique, for pre-test. The conventional groups adjusted 
mean was 65.8 percent compared to 71.4 and 69.9 
percent for the machine and book groups. 

Considering variability, however, the data indicate 
that the variance of post-test scores for each of the pro- 
grammed instruction groups was significantly larger 
than that of the conventional group (p < .05). How- 
ever, this was true for the pre-test also. Comparing 
pre- and post-test variances for each group shows that 
in no case was there a significant difference between 
variability before and after instruction. While the ob- 
tained differences are in the direction of an increase in 
variance after instruction for all three groups, this in- 
crease was not significant. 

It is interesting to note that these data do not support 
the hypothesis that programmed instruction reduces 
variability in achievement scores. Claims have been 
made that programmed instruction is the great educa- 
tional equalizer. In theory, this appears to be a com- 
pletely reasonable In a conventional class, all 

individuals are exposed to the same material for the 
same length of time. Because of the individual differ- 
ences of the students in understanding and mastering 
what they have been exposed to, the variability of pro- 
ficiency scores should be directly proportional to the 



Means and Variability of Achievement Test Scores 





Pre- 

Test 

% 


Post- 

Test 

% 


Adjusted 

Post-Test 

% 


Gain* 

% 


Conventional 

Mean 


47.93 


67.86 


65.82 


38.67 


Standard 

Deviation 


5.87 


7.82 




10.79 


Range 


36-59 


49-79 




11-53 


Machine 

Mean 


46.67 


72.32 


71.40 


49.93 


Standard 

Deviation 


13.03 


13.44 




16.02 


Range 


26-71 


43-91 




17-79 


Book 

Mean 


42.99 


67.56 


69.89 


45.24 


Standard 

Deviation 


12.10 


15.40 




19.70 


Range 


28-68 


42-90 




3-72 


Post-pre 

♦Gain = X 100 

100-pre 

Table 1 
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variability of these individual differences in assimilation. 

In the experimental group, under the conditions in 
which our experiment was conducted, the problem of 
time competition of individuals might have tended to 
overshadow the importance of proficiency and artificially 
reduced time variability. Each man could readily tell 
how much progress the other men had made with the 
course in relation to his own. Few men care to finis h 
last. Therefore, the subjects may have taken time only 
to partially master the m.aterial in an attempt to keep 
pace with the class. Perhaps if each student was per- 
mitted to take his course completely isolated from his 
classmates this competitive factor would be circum- 
vented and the time-proficiency-variability relationship 
hypothesis supported. 

Many assumptions as to why variability was not re- 
duced with programmed instruction could be debated, 
but unfortunately we have no facts to support any at 
this time. In any event, it is an interesting area deserv- 
ing experimental investigation. 

In the comparison of times for the groups, measures 
of time are of the time taken to complete the course 
once. Breaks are included but lunchtime is not. The 
time spent in classroom instruction for the conventional 
group was fixed at nmeteen hours for all subjects. The 
nineteen hours, however, did not include any home 
study which was known to have occurred for many 
students. Since there was no control or accurate meas- 
ure for the extra time for the conventional group, the 
programmed instruction group’s times are compared to 
19 hours. 

Table 2 shows the average times taken by subjects 
using machines and books were 14.53 and 13.64 hours, 
respectively. Thus, the machine group showed a 24% 
time savings and book group 28%. The difference be- 
tween each of the programmed instruction groups and 
the classroom group are statistically significant (p < 
.01). The difference between the programmed groups 
is not significant. 

In summary, we found that for our selected 19 hour 
course, “F’lndamentals of Computers,” personnel using 



Time Used by Conventional and Programmed 
Instruction Groups 

Conven- 





tional 


Machine 


Book 


Mean Hours 


19 


14.53* 


13.64* 


Time Savings 
Standard 


— 


24% 


28% 


Deviation 


— 


2.66 hours 


3.50 hours 


Range 


— 


7.20-21.10 


8.47-19.72 



* Significantly less than 19 (p < .01). 

Table 2 
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the branching program completed the instruction in 
significantly less time than did the classroom group and 
that they did so without reduction in proficiency. The 
obtained non-significant differences between the three 
groups in achievement were in favor of the programmed 
course. Finally, it was noted that although some vari- 
ability of time did occur with programmed instruction, 
no decrease in the proficiency variability resulted. 

Using the same group of subjects a further analysis 
of the data was made to determine if there were differ- 
ences in the way in which use was made of time depend- 
ing upon whether the mode of instruction was pro- 
grammed or conventional. The student of today is 
educated in an environment where the fifty-minute hour 
is the widely accepted classroom work-rest cycle. In 
this environment he is exposed to the material he is to 
master for fifty minutes of each hour. He is then given 
ten minutes free time to use as he sees fit. He repeats 
this cycle until that “magic hour” when he has been 
exposed to all the material determined as essential to 
the particular course. 

Administratively, then, the problems of handling per- 
sonnel are kept to a minimum . A class starts on day X 
and finishes on day Y, a day known even before day X 
begins. All men of the selected class can be treated as 
a single entity. Scheduling is made relatively easy; per- 
sonnel loads and teacher availability can be more read- 
ily matched; classroom needs can be better predicted, 
and so on. 

But the administrative problems are not the only 
problems which must be solved. In reality, the primary 
problem which should be solved in a learning situation 
is the problem of teaching and student learning. 

Unfortunately, the two seem to work against each 
other. The establishment of a learning environment 
which minimizes the administrative problems tends to 
magnify the problems or establish roadblocks in the 
learning process. This occurs because the material to 
be mastered must then be doled out to the individuals 
at a pace predetermined by those training specialists 
who developed the plan of instruction (POI) and then 
this pace is further modified to suit the need of the 
instructor or trainer who is responsible for transmitting 
the information to the students. 

This established time element, when coupled with 
the individual differences of the students, results in one 
of a number of things occurring. First, the established 
time can be set or adjusted to the intellectually low- 
average element within the class. This usually results 
in having most of the class pass the course, but it ac- 
complishes this at the sacrifice of, and to the detriment 
of, the intellectually bright or fast student. Or second, 
the established time can be adjusted to the average 
within the class. But this does two things in that either 
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fewer persons master the material, and pass the course, 
or that more students must spend substantially more 
time in home study just trying to keep abreast of the 
class. However, it is generally found that this home 
study is required by those who are in need of the 
intellectual intercourse normally provided by the in- 
structor, which is a need tliat cannot be satisfied in 
home study. 

Devotees of programmed instruction claim that one 
of the major benefits of the application of this new 
technology to the learning environment is the self- 
pacing feature which allows each student to proceed 
along at his own pace through the material. In this 
manner each student of every class is exposed to the 
same material and is neither speeded up beyond his 
capabilities nor required to drag-his-heels because of 
his slower classmates. But immediately the administra- 
tive problem is brought up and the question as to how 
will we be able to handle these people administratively 
is sounded throughout the land. 

While this paper does not intend to cover the entire 
problem of the administrative-learning relationship, it 
does seek to discuss one of the probable causes of the 
many faceted administrative complex relating to stu- 
dent self-pacing or student control of his own time. 

What does happen when a student is given permission 
to regulate his study and break needs as he feels they 
should be so that he may best learn the material? Our 
concern with this area was to determine i£ this self- 
regulation would result in the students’ actually taking 
more clock-time to go through the course than the nine- 
teen hour limit, 2 eight hour days and 3 hours, allo- 
cated for the classroom learning of our selected course. 

It was felt that a time increase could result from the 
students’ taking excessive breaks, extended lunch hours, 
working shortened days, or any combination of these 
factors, and that this behavior would be due to one or 
two factors. First, the student would require longer 
bre^'Iis and shorter exposure time now that the learning 
process was entirely through his reading the material 
rather than through the media of an instructor, thus 
resulting in increased fatigue on the part of the student. 
Secondly, it was felt that with this new found freedom, 
some of the personnel would tend to take advantage of 
the situation and take more break time than they really 
needed, just because it was now available to them. 

In an experiment, designed to compare conventional 
classroom with programmed instruction with regard to 
time and ability to teach, the students who were taught 
by programmed instruction were allowed to pace them- 
selves and control their own work-rest cycles. They 
were told specifically that they were to take as many 
breaks as they felt they needed, of as long or short a 
duration as they felt necessary. The conventional group 



(N = 15) operated in the normal fifty-minute hour 
work-rest cycle. 

At this point a difference in data collection of the 
two groups should be noted. One experimental group 
of 30 subjects using Mark II Autotutors, and another 
consisting of 25 students using a programmed text were 
timed to the nearest minutes for all breaks taken. The 
procedure used was to tally a break time anytime the 
student left his assigned desk and to mark the break 
concluded upon his return. For the conventional group, 
no such timing procedures was used; break time was 
based only on the accepted classroom schedule which 
called for a ten minute break every fifty minutes. 

The comparison of breaking behavior of the three 
groups is best seen in Table 3. 

The conventional group, as controlled by the fifty- 
minute hour, took the normal thirty minutes each morn- 
ing and afternoon as brought about by the three ten 
minute breaks each session. The reason for only three 
ten mmute breaks each session for the conventional 
group was the overall daily schedule by which they 
abided. Table 4. The similarity of the two experimen- 
tal groups in their breaking habits is readily depicted 
by the first three columns of Table 3. Both groups 
averaged more breaks per man in the morning sessions 
than were taken in the afternoon sessions and the time 
per break averaged slightly longer in the mornings than 
for the afternoons. In all cases the average number of 
breaks was less than that for the conventional group. 

Because of the freedom of self-pacing allowed the 
programmed instruction group over that of the class- 
room group, it was thought that the reduction in num- 
ber of breaks taken could be caused by personnel 
taking extended break times, extended lunch hours, 
starting late, or leaving early on each day thus resulting 
in their not needing as many breaks because they were 
not there as long as the accepted eight-hour day. 





Break 


Data 

Break 
Time/ Time/ 


Single 

Break 




Breaks/ Break 


Man 


Range 




Man 


(Min) 


(Min) 


(Min) 


Conventional 


Mornings 


3 


10 


30 


— 


Afternoons 


3 


10 


30 


— 


Machine 


Mornings 


1.24 


15 


19 


1-57 


Afternoons 


1.12 


13 


15 


1-87 


Books 


Mornings 


1.26 


19 


23 


1-63 


Afternoons 


1.07 14 

Table 3 


15 


1-42 
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The distribution of the time-per-break data is shown 
in Table 5. Because so few of the personnel took 
extended breaks it seems safe to assume that the in- 
fluence of these few on the number of breaks needed 
woud be negligible. Also it may be seen in Break 
Time/Man column of Table 3 that extended break 
times were not predominant in their occurrences. 

In looking at the lunch hour comparison, Table 6, 
it is seen that the average time taken for aU student 
lunches is only six minutes longer for the machine 
group and one minute less for the book group than 
was normally allotted these people. 

A detailed analysis of the “excessive” lunch hour 
data revealed that there were no real violators in the 
book group and that only four men took excessive 
lunch hours in the machine group. In each of these 
cases it is readily stated that these persons were in a 
position to t^e the end of the course in sight and thus 
were able to take more time for handling of personal 
business or just to enjoy the luxury of a long lunch. 

One excessive lunch, 132 minutes, was taken when 
a man finished lesson 5 at 1030 in the morning. With 
one more lesson to go, the entire course was divided 
into six lessons, he left and returned at 1242. 



Daily Schedule 



1st Period 


0730-0820 


2nd Period 


0830-0920 


3rd Period 


0930-1020 


4th Period 


1030-1120 


Lunch 


1120-1240 


5th Period 


1240-1330 


6th Period 


1340-1430 


7th Period 


1440-1530 


8th Period 


1540-1630 



Table 4 



Break Length 





Machine 
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Book 
Groups 


Classroom 

Group 


1- 5 minutes 


28% 


22% 


— 


6-10 minutes 


16% 


17% 


— 


10 minutes 


— 


— 


100% 


11-15 minutes 


19% 


14% 




16-20 minutes 


17% 


11% 




21-25 minutes 


8% 


18% 




26-30 minutes 


4% 


7% 




31-60 minutes 


7% 


10% 




60 minutes 


.6% 


.8% 
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Lunch 










Time 


Range 




N* 


(Min) 


(Min) 


1st Day 


15 


80 


— 


2nd Day 


15 


80 


— 


3rd Day 


— 


— 


— 


AU 




80 




Machine 








1st Day 


29 


83 


33-179 


2nd Day 


15 


94 


75-132 


3rd Day 


1 


67 


— 


AU 




86 




Book 








1st Day 


23 


85 


84-94 


2nd Day 


12 


68 


43-88 


3rd Day 


— 


79 




* Varying Ns for machine and book groups was 


brought 


about by personnel finishing course in varying times, due 


to self-pacing, or by taking an entire halfday session off. 


again a self-pacing feature, but 


one which also occurs 


with students in 


conventional classes. 






Table 6 







A second extended lunch, 179 minutes, occurred 
when the man was already on lesson 6. 

The third extended lunch, 178 minutes, occurred 
from 0932 to 1230 and this man was on lesson 5. 

The fourth extended lunch, 114 minutes, occurred 
with the man on lesson 6. 

In each of these instances the total time for these 
men to finish the nineteen hour conventional course 
was less than nineteen hours and in some cases was 
less than the average of the experimental groups. 

It might be said, then, that extended lunch hours 
might have had some influence on the number of 
breaks taken by the experimental groups, but that this 
influence would have been negligible. Because only 
four persons took long lunches, it is felt that, had they 
conformed to the 80 minute lunch, they each might 
have taken one more break m the morning, but in 
the aggregate this would not have caused the 1.24 
breaks per man figure to approach the conventional 
three breaks per man. 

The third influence on breaks might well have been 
the student’s freedom to select his own starting and 
finishing time. Detailed analyses of this data showed 
that in the machine group aU personnel were in before 
0800 (0730 normal start time) each day, and that for 
the book group only one man came in after 08(K) 
(0805), and two men worked only a half day each 
for one of the days. 
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For the\ afternoon leaving time, a different picture 
does appear which tends to justify the lower “number 
of breaks” values shown in Table 3 for the afternoon 
sessions. Normal departure time was 1630 hours. For 
the book group this was generally adhered to. How- 
ever, on the first afternoon three persons (N = 24) 
left before 1620 (1435, 1534, and 1546 hours) and 
on the second afternoon six (N = 21) left before 1620 
hours. What is interesting to note is that six out of 
the nine departures on the two days were persons who 
had just completed a lesson and did so within 45 min- 
utes of normal quitting time. It was a good stopping 
point. For the other three, no justification is apparent 
from the records. 

In the machine group the same phenomena appeared. 
On the first afternoon, five left early and four of these 
had just completed a lesson. On the second afternoon, 
fifteen left early with ten being at a known convenient 
break point. 

Because of these early departures, it is probable that, 
had all personnel been required to stay until 1630 
hours, the 1.12 and 1.06 breaks per man averaged in 
the afternoon might have been brought up to the 1.24 
and 1.26 level average of the morning. It is doubtful 
that it would have brought the average up to the con- 
ventional three per session. 

It is interesting to note that, of the 55 experimental 
subjects, only four exceeded nineteen calendar hours 
and that of these, three required in excess of nineteen 
hours exposure just to learn the material. The fourth 
required eighteen hours and 59 minutes to learn the 
material, or one minute less than nineteen hours. 

Thus it appears as though the concern related to stu- 
dents exceeding a conventional time standard by con- 
trolling their own work-rest cycles in a learning en- 
vironment is, on the whole, not supported. Students 
using programmed instruction actually took fewer 
breaks for any given time session than did their class- 
room counterparts. While they took longer break peri- 
ods, the total break time per man per day, or half day 
session, was substantially less than the time allowable 
on an enforced 50 minute hour pattern. 

It should also be noted that the difference from the 
conventional schedule in time per break. Table 3, may 
be only an apparent difference due to the procedural 
techniques of measuring the break time of the pro- 
grammed instruction students while simultaneously 
merely accepting the “take-ten” break time of the class- 
room group. When one recognized the fact that the 
conventional ten minute break often exceeds an actual 
ten minutes time, then it might even be concluded that 
programmed instruction groups took less than half as 



many breaks and took no more time per break than 
they would have been allowed in a normal military 
learning environment. 

It is felt, however, that the behavior patterns mani- 
fested by this experimental group can lead to one or 
two possible assumptions. The first, that the breaking 
habits noted here were notably different than those to 
which the subjects were normally exposed because of 
the novelty and newness of the teaching or educating 
mechanisms. Because the students found it necessary 
to work only an average of fourteen hours before they 
felt that they had attained mastery of the material, it 
can be theorized that the exposure time was sufficiently 
short so that the so-called “pin ball” effect had not had 
an opportunity to dissipate. This then raises the ques- 
tion, “Would continued exposure and/or familiarity 
with these new programmed instruction techniques 
result in different breaking or work-rest patterns?” 

The second, and perhaps more far reaching, assump- 
tion is the one which says that the work-rest cycles 
exhibited during this experiment are true indications 
of what students need in order to optimize learning. 
Perhaps the behavior elicited by those students who 
were able to control their own breaking patterns, fewer 
breaks of slightly longer duration, does really suit the 
needs of more people, and perhaps should even be 
investigated for potential application to classroom use. 

Perhaps 50 minutes, while sufficient to warmup to 
the learning task, is not long enough to realize maxi- 
mum benefit of the warmup period so essential to the 
overall learning process. Perhaps the problem of hav- 
ing to stop and start three times each morning and 
each afternoon results in less actual leaning than 
would occur if the students took less breaks of longer 
durations during any given learning environment. Un- 
fortunately our experiment can provide no answer here, 
but it does raise the question for speculation. 

In summary, then, our investigation of work-rest 
cycles for programmed instruction students showed that 
these students took substantially fewer breaks of slightly 
longer duration than did their conventional counter- 
parts. In addition, we found that students who took 
extended lunch hours, or who worked shortened days, 
did so because they were near the completion of the 
course, or because they had completed a lesson near 
quitting time and took the lesson completion as a good 
stopping point. It was also found that students who 
required more than the conventional time to finish the 
course materials took less breaks than their conven- 
tional counterparts, and that these breaks were also of 
a shorter duration. 
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THE UNINVITED 



Lois-ellin Oatta 

This author discusses certain very real dangers that programmed instruction presents. 
The rigorous and specific development of course objectives might well reinforce 
"uninvited guests" inadvertently included in the program, as well as attitudes and 
perceptions that will emphasize rigid, dogmatic inflexibility instead of the capable, 
freely inquiring minds that are our objectives. She reports data that indicate that 
dogmatic thinkers apparently interest more effectively with linear programs than do 
less dogmatic students. The implications are thought provoking. 



Since confession is supposed to be good for the soul, 
I have a confession to make. I used to be a conven- 
tional teacher, and I’ve carried some of the interests 
from my conventional days into programmed instruc- 
tion. I would like to share with you one of these con- 
cerns — am going to talk about authoritarianism and 
programmed instruction. I will begin by showing how 
such uninvited guests may enter when we open the door 
for course content. In the second part of the discus- 
sion, ru review the meaning of the authoritarian com- 
plex of fascism, rigidity, and dogmatism. Then I’ll pre- 
sent evidence we discovered which indicates that a typi- 
cal programmed course may be reinforcing dogmatism 
along with diodes. This evidence will be discussed in 
detail later: to put it bluntly, dogmatic students did bet- 
ter in a programmed course. My guess as to the “whys” 
is a bit of a long story, but I hope you will bear with 
me because the ethical problems raised by the data are 
significant and are likely to be with us for a long time. 

My first point is that while we know that a valid 
programmed course teaches the goal behaviors, we do 
not know what else may be learned at the same time. 
As I was leafing through “The Analysis of Behavior’’ 
the other day, I came upon a list. It read, in part, 
“Reduce alcoholism. Increase the nutritional adequacy 
of the average American diet, and demonstrate the con- 
tribution of formal logic to creativity.” Sounds pretty 
varied, yet these have all been given, and recently, too, 
as goals of education. The list is a sort of Exhibit A 
for one of the five ways in which the conventional 
course differs from programmed instruction. The edu- 
cational goals of the conventional course are usually 
many and fuzzy. It would be hard to tell from some 
of the objectives whether you were looking at a design 
for a course in physics or one in home economics; 
either — or neither — could reduce alcoholism and touch 
on creativity. The educational objectives of a pro- 
grammed course, a the other hand, MUST be specified 
in clear behavioral terms, and the program is designed 
to teach just these observable goal behaviors. We say, 
in fact, that clear specification of the goal behavior is 
one of the benefits programmed instruction has con- 
ferred on conventional education. We believe that 
teachers who use the analytic frame-work of pro- 



grammed instruction are more likely to get where they 
want to go because they now know where they're going. 
And we know, of course, that programmed instruction 
can provide the best road. 

After a programmed course on extraction of the 
square root, Susie can extract a square root — and that’s 
that. Or is it? It seems to me that we may have fallen 
into a fallacy here which I think logicians call the 
Pathetic Fallacy, ^and it is pretty pathetic. A valid 
programmed course by definition is one through which 
the student achieves the initially specified goal be- 
haviors and many programs succeed in being valid with 
admirable consistency. But we do NOT guarantee that 
these goal behaviors AND ONLY THESE behaviors 
are reinforced. And when we stop for a minute, it 
doesn’t seem likely that we ever can claim that only 
the course content is being taught. I do not think, as 
one example, we can separate reinforcing course con- 
tent from reinforcing the properties of the structure 
within which the content is presented. And the non- 
content extras can be valuable, neutral, or disastrous. I 
believe we have developed tunnel vision in our specifi- 
cation of very limited terminal behaviors. This seems 
to apply to knowledge of subject matter (we are, for 
example, probably reinforcing sentence structure as well 
as sentence content) and to more elusive changes. This 
is pretty obvious if we talk about conventional instruc- 
tion. A lot of kids have low opinions of the educational 
process, not just English I, when they quit school at 16. 
Now let’s put the shoe on the other foot — our foot. The 
usually positive attitude students express towards pro- 
grammed courses certainly suggests they are learning 
something about this kind of teaching technique — yet 
most of the programs I’ve seen haven’t listed, “Increase 
positive responses toward programmed instruction on 
attitude Scale I” as a goal behavior. The goals of the 
programmed courses do say very clearly what is wanted 
and if the program is good. I’ve found that the knowl- 
edge is there. But the goal specifications say nothing 
at all about what is not there or about what else the 
wind blew in when the door was open. 

My second point is that uninvited guests are chancy 
creatures. They are sometimes very pleasant. The 
story of the three princes of Serendip who had a genius 
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for finding v/hat they were not looking for has become 
very popular. But there is also the much older legend 
of Pandora’s box; quite uninvited evils were inside 
along with Hope. I am going to tell you about one 
such guest which may be hidden among the welcomed 
aspects of programmed instruction. This guest is part 
of the complex of authoritarian attitudes. li is not part 
of the content and I think it comes in with the structure. 

Let me review with you the meaning and measure- 
ment of the authoritarian complex. Authoritarian atti- 
tudes sounds like a gen-u-wine fright word and it is. 
The major study goes back to World War II, when the 
consequences of fascism were seared into the awareness 
of most of us here. The brutalities which men, civilized 
men, committed on other men — and women, and chil- 
dren — seemed to go far beyond a superficial attitude 
veneered on the German psyche by a demagogue. And 
some social scientists thought that underneath the many 
varieties of intolerant behavior was a single attitude 
structure of fascism. By fascism they meant boot- 
licking the boss, kicking the kids, and hating your 
neighbor, a basic personality structure developed from 
very early childhood experiences. These social scientists 
developed a measure for fascistic tendencies which they 
called the F scale. It included items such as, “obedi- 
ence and respect for authority are the most important 
virtues children can learn,” and- they found that sub- 
jects who were high in fascistic authoritarianism were 
indeed anti-Semitic, politically conservative and gener- 
ally ethnocentric. But other social scientists found that 
the F scale didn’t measure quite what it should, that 
while it reflected acceptance of far right social ideas, it 
failed to discriminate between the totalitarian attitude 
of the Communist, the intolerance and inflexibility of 
the professional liberal, and an open-minded approach 
to belief systems. I’ve used the term “belief system” here 
for the first time and I’ll be using it frequently from 
this point on. Let me define it. A belief system is all 
the beliefs, sets, expectancies or hypotheses that a per- 
son at a given time accepts as true about the world 
he lives in. Rokeach and his social scientist associates 
(1960) have succeeded in differentiating at least three 
levels of authoritarian beliefs. The F scale, they found, 
did indeed reflect the specific ideological content of the 
far-right but they discriminated at least two more ten- 
dencies. These tendencies are structural. That is, they 
reflect not the content of a belief system but how it is 
developed and maintained. The first more inclusive 
tendency is rigidity, which is defined as resistance to 
change within the elements of a belief system. A per- 
son with rigid educational expectations, for example, 
would be resistant to change in the specific operations 
of teaching. The chap with rigid educational expecta- 
tions would resist unscrewing the chairs from the class- 
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room rows and grouping them around the teacher. An 
example of an R scale item is, “I am often the last one 
to give up trying to do a thing.” And Rokeach found a 
still more general tendency which he called dogmatism. 
Dogmatism is resistance to ‘hange in belief systems 
by intellectual conviction. The highly dogmatic indi- 
vidual doesn’t balk at a change in the elements but has 
difficulty in resynthesizing them into a new and differ- 
ent belief system. He could, for example, accept mov- 
ing the chairs all over the map as long as the process 
of absorb-what-the-teacher-says was unchanged by the 
physical differences. An example of a D scale item is, 
“young people should not have too easy access to 
books which are likely to confuse them.” Rokeach 
describes a dogmatism continuum with a closed-minued 
side and an open-minded side. On the closed-mined 
side, he would identify extremists of all persuasions and 
on the open-minded side, he would identify individuals 
with a willingness to explore the evidence for a change 
in elements and the ability to synthesize them into a 
new belief-disbelief system, to restructure and evaluate 
freely. 

I have spoken at some length to define the authori- 
tarian complex. None of the characteristics about 
which I’m speaking — racial and religious intolerance, 
fascistic authoritarianism, rigidity and dogmatism — 
evokes a pleasant picture. They aren’t meant to be- 
cause the s-'cial situations with which they are asso- 
ciated can be pretty horrible. All in all, they sound like 
attitudes to be avoided. And my third point is that one 
of the frequently mentioned educational goals in the list 
which fluttered out of “The Analysis of Behavior” some 
minutes ago is avoiding closed-mindedness. The state- 
ment is more usually positive — to foster a critical and 
free understanding, to develop a way of thinking which 
can generalize beyond the specific subject matter to 
form an approach to other areas of knowledge and 
behavior. 

In the past 15 years, there has been an increasing 
concern about the development of the freely inquiring 
mind. The pocketbook sincerity of this concern was 
demonstrated when the Mellon Foundation financed in 
the 1950’s a very ambitious and extensive evaluation 
of the extent to which the aims of a liberal education 
were being reached. The Vassar study also yielded 
many interesting data, as such studies do. Among the 
data was the encouraging evidence that in conventional 
college courses, authoritarian tendencies were usually 
negatively related to achievement. This negative rela- 
tion had been previously reported, of course, with the 
F-scale and achievement, and in subsequent non-Vassar 
studies, scales of the authoritarian complex reassur- 
in^y continued to be negatively correlated with grades 
in high school and college. 
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Lois-ellin Datta 



You may be wondering what all this has to do with 
programmed instruction. Nothing, one would wish, or 
at leas<^, the negative correlation betv'een achievement 
and closed-mindness should continue to be reported for 
programmed courses. Few programmers would hope 
they are reinforcing dogmatism, rigidity, and prejudice 
along with information about reading circuit diagrams. 
But in point of fact, we think that in its typical use, 
authoritarian attitudes and programmed instruction may 
have a lot to do with each other, for we found a high 
positive correlation between dogmatism and achieve- 
ment in a linear programmed course administered in a 
fairly typical manner. 

Let me tell you what we did, what we found and 
what we think it all may mean. We are Education and 
Training personnel at the Missile and Space Division 
of the General Electric Company. The Missile and 
Space Division is responsible for research and develop- 
ment of re-entry and orbital systems and we are respon- 
sible for supporting this effort by appropriate educa- 
tional activities such as in-plant courses. After reading 
the articles in Look and The Reporter, and listening to 
some of you who are here tonight, it seemed to us that 
programmed instruction, under optimum conditions, 
has tremendous promise for education, and what were 
we waiting for? The trouble was that no one could tell 
us just what these optimum conditions were, and the 
few systematic studies such as Roe’s (1960) that we 
found suggested that either there were no optimum 
programming conditions or they hadn’t yet been found. 
Undaunted, we’ve been trying to search systematically 
in other pastures for optimum conditions of student, 
administration and criterion variables. (Datta, 1963) 

In one such study, we were field-testing a pro- 
grammed course in solid state electronics. The three- 
section course is a 3000 frame linear program; response 
was overt and a consulting instructor mode of adminis- 
tration was used. Tlie consulting instructor met the 
students six times for testing and to answer their ques- 
tions; he was available ad libitum in between to answer 
questions but he didn’t take the initiative in leading a 
discussion or suggesting supplementary material. I'hree 
groups oi' Missile and Space Division personnel partici- 
pated in the study. They were all applicants for the 
in-plant, after-hours, voluntary education program. All 
had a speaking knowledge of electronics and ail swore 
they really needed the course for their jobs. The groups 
varied in pacing but since pacing did not affect achieve- 
ment and since the effects which I’m leading up to v/ere 
replicated, the data I’m going to present are for 30 
pooled Ss. The men submitted to a few tests, including 
the three scales of the authoritarian complex, a measure 
of general mental ability, a test of basic electronic 
theory, precourse knowledge of transistors and finally, 



three post-tests, one for each section. The criterion 
achievement tests were NOT in any sense test frames. 
They were developed by the instructor on the basis of 
course content to measure what a chap who claimed 
knowledge in the content areas ought to be able to do 
regardless of how he was trained. The split-half relia- 
bilities of all three measures were high and content 
validity was judged satisfactory by tne program authors. 

Now that I’ve told you who we are and what we did, 
let’s consider what we found. The results are shown on 
this chart (see Table I) as the correlations between the 
three authoritarian complex scales and achievement on 
post-course test I and as the percentage of subjects 
high, average and low on each scale who were above 
the median on post-course achievement. 

The correlation of F scale performance and course 
achievement, —.17, was zero-order. 80% of the stu- 
dents in the lower third, 60% of the students in the 
middle third, and 20% of the students in the upper 
third of the F distribution were above the median in 
course performance. 

The correlation of R scale scores and post-uourse 
test I achievement v/as —.42. The percentages of stu- 
dents in the lower, middle and upper thirds of the R 
distribution who achieved scores above the median on 
test I are 90%, 60%, and 10%, respectively. The 
correlation of D scores and achievement is -j-.53; 70% 
of the high D scorers, 80% of the middle D scorers 
and 10% of the low D scorers were above the group 
median on test I. 

The results are consistent with Rokeach’s differentia- 
tion and are, to us, quite thought-provoking. The low 
correlation between the F scale and achievement prob- 
ably reflects the lack of political content in a course on 
solid state electronics. 

We are not at all sure what accounts for the latter 

Table I 

Percentage of Students in the Upper, Middle, and 

i.2/'ver Thirds of Attitude Scale Distributions Above 
the Median on Post-Course Test I 



Scale 


Upper 


Middle 


Lower 


Fascism 


20% 


60% 


90% 


Rigidity 


10% 


60% 


80% 


Dogmatism 


70% 


80% 


10% 


Rank-Order 


Correlation 


of Attitude 


Scales and 



Post-Course Test I Performance 





RHO 


F/ Achievement 


-.17 


R/ Achievement 


-.42 


D/ Achievement 


4- .53 
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two relationships. One explanation is that the educa- 
tional expectations of the high D students are consistent 
with the structure of linear programmed courses admin- 
istered as this one was while the educational belief sys- 
tems of the low D students are consistent with the more 
inductive approach traditionally found in in-plant 
courses. One interpretation of the negative R/ achieve- 
ment relation is that the open-minded low R student 
adjusts rapidly to differences in the elements of pro- 
grammed and conventional instruction but the high R 
students’ greater resistance to change is at least iru- 
tially a handicap. One would expect on the basis of 
these analyses that if you asked the students to describe 
the “typical” course and the programmed course, you’d 
find two kinds of differences. The first kind of differ- 
ence is qualitative. Students high in R would describe 
structural elements of the educational system. Students 
high in D would describe the system itself. The second 
kind of difference is quantitative. Students high in R 
would probably have 180 ° between perceived typical 
and programmed course characteristics. Students high 
in D would have congruent lists for typical and pro- 
grammed courses while for students low in D, the lists 
would not agree. We are currently conducting several 
follow-up studies, measuring the authoritarianism/ 
achievement relation in traditional in-plant courses, the 
effects of educational expectations, of fact vs. problem- 
solving criteria, and so on. 

The point I am raising here is that while the pro- 
grammer has an obvious charge to meet the course con- 
tent objectives, we might also consider our responsibil- 
ity to determine what else besides the stated behavorial 



objectives is being reinforced throu^ course structure 
and administration. In an in-plant course on solid 
state electronics, it mattereth little in the ultimate 
scheme of things that students high in dogmatism seem 
to have an advantage, for lo, almost all survivors re- 
ceive certificates of completion. But in systems where 
achievement is related to who gets the best job, who 
gets the chance to go to college or to graduate school, 
the problem of the uninvited guest becomes a serious 
ethical issue. We hope that programmed instruction 
will be a real revolution in education, that it will help 
us reach the “Promised Land” where school drop-outs 
are reduced, where the level of knowledge is uniformly 
increased, and where achievement is independent of 
general mental ability or socio-cultural inequities. We 
may indeed find All of this to be true — ^but also find 
homogeneity of thought, conformity, and dogmatism; 
we may discover that we have educated a nation of 
walking computers. 

It would indeed be premature to cry “Flood” from 
one drop of rain . . . and I certainly don’t predict that 
a nation of walking computers is any more an inevitable 
concomitant of programmed instruction than I would 
claim that a nation of freely inquiring minds is the 
inevitable result of all conventional courses in all seg- 
ments of this country. But I am wondering if these 
concerns are no less ours than theirs and I hope that I 
have rather suggested the challenge of developing pro- 
gramming techniques and methods of utilization which 
will invite wisdom to join the guest which we have 
already welcomed, knowledge. 
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PROGRAMMED INSTRUCTION OVER CLOSED CIRCUIT TELEVISION 

Walter Dick 

Walter Dick, reporting on research conducted by C. R. Carpenter and L. P. Greenhill 
at Pennsylvania State University, suggests that considerable variation can be intro- 
duced in the application of programmed instruction, in fact, even seif-pacing is ques- 
tioned. The use of programmed materials on closed circuit television was studied with 
results that open vast possibilities for training and education amplification. 



The general purpose of this series of research studies 
was to investigate the possibilities of programming 
entire courses in mathematics and English grammar 
and to compare different methods and media, with spe- 
cial emphasis on closed-circuit television, for presenting 
programmed learning materials. In order to accomplish 
the objectives it was necessary to analyze especially the 
problems of pacing and individual and group conditions 
for the learning of programmed materials. 

The research dealt specifically with the following 
subjects and problems: (1) A critical analysis of some 
assumptions of programmed learning and implications 
for adapting and using the new media; (2) The devel- 
opment of programmed materials for research purposes; 
(3) Experimental comparisons of external-pacing and 
individual-pacing for the study of programmed mate- 
rials using different methods and media for presenta- 
tions; (4) Comparisons of different rates of pacing and 
of individual and group conditions of study; (5) Com- 
parisons of programmed televised presentations with 
(a) teaching machine presentations, and (b) instructor 
taught classes; (6) Studies of the effects of the mini- 
mum group of two students studying programs in com- 
parison with individual study; and (7) Preliminary 
analyses of quantitative and verbal abilities of students 
relative tc achievements using programmed courses, 
and of a personality factor relative to these study 
conditions. 

The content areas selected for programming were 
introductory college algebra and English grammar. 
Special programmed courses were written, tested, re- 
vised, and adapted for presentation by teaching ma- 
chines, programmed textbooks, film strips, and 
closed-circuit television. The completed algebra course 
consisted of 30 units of instruction; the course on 
English grammar contained 14 units. Appropriate and 
reliable tests of learning were also constructed. In 
general, these included tests on each unit of the pro- 
grammed courses, mid-course tests, and end-of-course 
tests. Additional special purpose tests were also used 
in the studies. 

Six separate controlled experiments were conducted. 
Four of these used the algebra program and two used 
the English grammar program. These experiments are 
briefly summarized: Experiment I, contemporary alge- 
bra course, made comparisons between several methods 



of presenting the programmed course — ^by teaching 
machines (self-paced), by programmed textbooks 
(self-paced), and by film strips (externally-paced). 
Performance of students receiving the programmed 
instruction was compared with that of students receiv- 
ing direct instruction from an experienced teacher who 
covered the same topics by the lecture-discussion 
method. 

Results showed no substantial differences in per- 
formance among the three programmed treatments as 
measured by the sum of students’ scores on the unit- 
tests and their scores on a comprehensive end-of-course 
test. However, on the unit-tests the three programmed 
treatments combined produced significantly higher 
scores and lower variance than did the direct instruc- 
tion by a good teacher. On the end-of-course test the 
means were not significantly different, but again the 
variance for the programmed treatments was signifi- 
cantly lower than that for the direct instruction. 

Experiment II investigated the effects of varying the 
pacing of externally-paced programmed materials pre- 
sented to groups of students by means of film strips. 
The initial times for each frame under the externally- 
paced conditions were based on the mean time taken 
for each frame by a sample group of smdents working 
on a self-paced schedule. In this experiment these base 
times were referred to as 100%, and the groups were 
paced frame-by-frame through the programmed course 
using times for each frame which corresponded to 
80%, 90%, 100% and 110% of these base rate times. 

Results showed that within this range of variations 
in external-pacing there were no significant differences 
in student performances or in attitudes "toward pro- 
grammed learning. Again, externally-paced and self- 
paced study conditions were found to be equally 
effective. 

Experiment III involved a comparison between an 
externally-paced presentation of tlie programmed alge- 
bra course using closed-circuit television and a self- 
paced presentation using teaching machines. Results 
showed equivalent learning from both procedures as 
measured by summed unit-test scores and performance 
on mid-course and end-of-course tests. 

Experiment IV investigated another aspect of the 
pacing and grouping problem — namely, the pairing of 
students for the study of programmed materials. In 
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this experiment half of the students were randomly 
assigned to pairs; the remainder of the class worked 
as individuals. It was hypothesized that interaction 
among the students in each pair might facilitate learn- 
ing, and provide more favorable motivation, including 
immediate social reinforcement, for programmed in- 
struction. 

Results showed that the performance of the students 
in the two treatments was equivalent on the unit-tests, 
the mid-course and the end-of-course tests. Further 
analysis of the results indicated that different learning 
abilities affected performance in the two different treat- 
ments. In the paired group, verbal ability accounted for 
a substantially higher proportion of the variance than 
did quantitative ability, whereas in the group that 
studied individually the reverse was found. Student 
attitudes toward programmed instruction were neutral 
and not significantly different for the two groups. 

Experiment V which was conducted with the English 
grammar course comprised two studies with two dif- 
ferent populations. One of these involved a comparison 
between a closed-circuit television (externally-paced) 
presentation and a teaching machine (self-paced) 
presentation. 

The second study compared the performance of stu- 
dents taught by means of the programmed course pre- 
sented over closed-circuit television, with that of stu- 
dents who were taught the same subject matter by an 
experienced instructor who used the lecture-discussion 
method. 

The results showed that in each study there were no 
substantial differences in students’ performances as a 
result of the different treatments they received. How- 
ever, in the second study, student attitudes as measured 
by an attitude scale were significantly in favor of the 
instructor presentation. 

Experiment VI was designed to investigate a further 
variation in conditions of learning which involved the 
pairing of a programmed course. A preliminary attempt 
was made to relate a personality characteristic of stu- 
dents to these conditions. In this experiment students 
were randomly assigned to two groups. They were 
paired on a personality variable of dominance-submis- 
siveness. One group was paired so that each member 
of the pair was similar in this personality characteristic, 
i.e., the mosi dominant was paired with the next most 



dominant and so forth through the group. In the sec- 
ond group students were paired who were dissimilar 
on this variable. In this group students were ranked 
from most dominant to least dominant, and in a group 
of 30, number 1 was then paired with number 16 and 
so forth. 

Results showed no significant differences in the learn- 
ing of the two groups on the several tests that were 
used. However, there was some evidence that differ- 
ential abilities and characteristics of the learners con- 
tributed to performance in the two groups. 

It appears to be feasible to adapt and use those 
media which require externally-paced methods for pre- 
senting programmed materials to relatively homo- 
geneous groups of learners without adversely affecting 
their achievement. Students of tlie populations sample 
have a relatively wide tolerance for variations in pacing- 
rate. 

Closed-circuit television supplemented with printed 
materials can be used effectivelv' for the presentation 
of programmed instructional materials to groups of 
students. This opens up possibilities for the wide dis- 
tribution of programmed instruction, and for the inte- 
gration of programmed sequences with lecture-demon- 
strations in televised instruction. The use of pro- 
grammed materials on television, supplemented with 
printed response forms, provides an excellent means 
for obtaining active participation on the part of all 
learners and for providing immediate reinforcement of 
their responses. Results on the pairing of students to 
study programmed materials yield equal results as com- 
pared with individual study methods. The effects of 
this pairing procedure on motivation and attitudes are 
not yet conclusive and require further investigation. 

Sampling of two content areas, mathematics and 
English grammar, yielded essentially the same results 
on all variables which were experimentally studied. 

In general, this research challenges the classical as- 
sumptions that the best method of using programmed 
materials is that of self-pacing and individualized study. 
In other words, a learner’s own speed may not be the 
optimum rate. The research shows some of the possi- 
bilities of using the new media and combinations of 
media with paced programs for instructing large num- 
bers of students simultaneously in time-limited periods 
of instruction. 
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THE REQUEST FOR OVERT RESPONSES AS A FUNCTION OF 
RESPONSE RELEVANCE AND INFORMATION LEVEL* 

Lewis D. Eigen and Stuart Margulies 

The authors ore concerned with the problem of overt (written) vs. covert (thinking] 
modes of response. They find that the necessity for overt responding may depend on 
the difficulty of the response. If the response is a relatively unlikely one, measured in 
terms of information available to the student at the time of response, the overt mode 
leads to better learning. However, if the response is easy, or if it is incidental to the 
teaching in the frame, the mode of responding is not o significant factor. 



In several studies using programmed instructional 
material, students were instructed merely to “think” 
their responses instead of writing them. Results indi- 
cate that the students proceed through the program 
more rapidly than students who write their responses. 
Comparison of the post-test performance of students 
employing no written responses (“thought” or covert 
mode) with that of students who learned by making 
written (or overt) responses have yielded equivocal 
results. The present paper attempts to account for 
these divergent results by varying two response char- 
acteristics: the relevance of the required response to 
the material taught, and the information level of the 
required response. 

Information level is a response characteristic related 
to level of difficulty, but more readily quantified ob- 
jectively. Essentially, it is a measure of the “informa- 
tion” required to make a decision, and is computed as 
a function of the number of alternatives from which 
the decision is to be made. Information level was 
manipulated in the present study by constructing 
groups of three letter words of different informational 
value. The highest informational level was represented 
by words like ZBX, VRT, etc. in which three different 
consonants were combined. Since the first letter could 
be chosen from the 21 consonants of the alphabet, the 
second letter from the remaining 20 and the last letter 
from the remaining 19, the total information repre- 
sented by words like ZBX is dependent on a discrimi- 
nation between 21 x 20 x 19 choices. A lower infor- 
mational level was obtained by selecting three letter 
words like SIJ. These consonant-vowel-consonant 
words represent less information because their struc- 
ture is limited to 21 x 5 x 20 alternatives. A third 
category of words, CAT, FAT, BAT, MAT, HAT, 
etc., required far less information because selection 
could be limited to a very small number of words. 

In the present studies the difference between the 

* “The authors wish to express their appreciation to the 
Carnegie Corporation of New York whose financial assist- 
ance made this study possible.” 

“A more comprehensive treatment of the data referred to 
in this paper appears in The Journal of Programed In- 
struction, Volume II, No. 1. 



overt (written) response and the covert (unwritten) 
response was examined at each of the three informa- 
tion levels. 

In addition to the overt-covert response distinction, 
a second response characteristic investigated is the rele- 
vance of the respon-ses to the presented material. This 
may be referred to as the relevant-incidental learning 
variable. The example frame below is used to illustrate 
this dichotomy. 



Professor Fred Keller states that the formula 
for water is H 2 O. 

Therefore, a molecule of water contains 

atoms of hydrogen and atoms of oxy- 

gen. 



K we ask a post-test question dealing with the formula 
for water, we can regard this frame as one which elicits 
a relevant (or critical) response. If, on the other hand, 
the post-test question is structured to ask: “What is 

the first name of Professor Keller? ,” we 

can regard the example frame as providing only inci- 
dental learning, as measured by this post-test question. 

The relevant-incidental variable might be called the 
“exposure only” versus “exposure plus indicated re- 
sponse” variable. The efficiency of relevant versus in- 
cidental learning can be examined by constructing two 
post-test questions following frames such as the ex- 
ample frame cited. One such question would call for 
the formula of water, while the second question would 
require Professor Keller’s first name. If the students 
performed better on the former question, it could be 
concluded that relevant responses are more efficient. 
Conversely, if the latter question results in better stu- 
dent performance, this would be evidence of the greater 
efficiency of irrelevant responses. 

Design 

The plan of the study may be outlined as follows: 
each sltident is presented with an instruction booklet 
containing three items, each of which is presented four 
times. An item contains seven three-letter words, all 
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seven of which are drawn from the same information 
level. The three items represent samples from the low, 
intermediate and high information levels. Thus one 
item contains seven three-letter low information words, 
one item contains seven three-letter intermediate infor- 
mation words, and the third item contains seven three- 
letter high information words. 

Each item requires the reader to respond to three 
of the seven words but not to the remaining four. 
Repetitions of the item differ somewhat in format and 
cueing but always require a response of the same three 
words. The first presentation of each item allows the 
reader to copy the correct response, but subsequent 
presentations demand a more uncued response. Each 
presentation of each item is made on a separate page 
and is always followed on the next page by the correct 
response. Since three items were each repeated four 
times, and each item was followed by the correct re- 
sponse, the instructional booklet used contained 24 
pages. 

A portion of the instructional sequence used is given 
below. It shows both a cued and relatively uncued 
presentation of the intermediate information level item, 
one cued presentation of the low information item and 
one imcued presentation of the high information item. 
A straight line indicates a new page. 

Learn these nonsense words: 

RIZ TUV RUD TIB KIG CO J YOL 
Fill in the missing nonsense words: 

RIZ TUV COJ YOL 



RIZ TUV RUD TIB KIG COJ YOL 



RIZ TUV *** *** *** COJ YOL 
The missing nonsense words are — 
and 



RIZ TUV RUD TIB KIG COJ YOL 



Learn these words: 

BAT CAT FAT HAT MAT RAT SAT 

Fill in the missing words below: 

BAT CAT RAT SAT 



BAT CAT FAT HAT MAT RAT SAT 




ZBX VRT *** *** *** NSG YMQ 
The missing groups are 


and 


ZBX VRT MKL CFP DGJ NSG YMQ 





Three different instruction booklets were prepared. One 
introduced high information items first, one introduced 
intermediate items first, and one introduced low infor- 
mation items first. All contained 24 pages consisting 
of four presentations of each of three items plus the 
associated answers; only the order of presentation 
changed. 

Upon conclusion of the 24 page booklet, students 
were asked to complete a post-test examination. The 
examination required students to fill in all of the seven 
words contained in each of the three items. (Twenty- 
one responses were required.) Percent correct on the 
three selected words in each item is compared to per- 
cent correct on the remaining four words. This pro- 
vides a measure of relevant versus incidental learning 
(or synonymously, exposure only versus exposure plus 
indicated response.) This procedure is highly analogous 
to the application of separate post-test questions ask- 
ing for Professor KellePs first name or the formula for 
water in the earlier example. 

PROCEDURE 

362 students were drawn from grades 7 and 8 of 
two New Paltz Junior High Schools. These served as 
experimental subjects in the study, and were randomly 
divided into two experimental groups. One group con- 
tained 179 students, and was required to write its re- 
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sponses (overt response mode) in instruction booklets. 
The second group contained 183 students and was re- 
quired to think its responses (covert response mode). 
Except for the instruction which differentially instructed 
the two groups to “write” or “think” their responses, 
there was no other experimentally controlled difference 
between the groups. Both groups also were required 
to complete the same written post-test. 

Students took the post-test as soon as they com- 
pleted the instruction booklet. Typically subjects re- 
quired less than 10 minutes to complete both the in- 
struction booklet and the post-test. 

Results 

Figure 1 summarizes the results of the study. This 
data, when subjected to non-parametric analyses, sup- 
ports the following conclusion. 

(1) Performance scores decrease as informational 
level increases (or, almost equivalently, that students 
perform more poorly when the material is harder). 
This result is obtained whether the required response 
is covert or overt, relevant or irrelevant. (Friedman 
Analysis of Variance significant beyond the .001 level.) 

(2) Student performance is superior with relevant 
(critical) responses than with incidental responses. 
This result holds for all three information levels 
whether the response be made overtly or covertly. (Sign 
test significant beyond the .001 level.) 
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(3) Overt responding yields more superior student 
performance than covert responding for intermediate 
and high information levels when the responses are rele- 
vant. (Mann-Whitley U Test is significant beyond the 
.0001 level.) No significant differences, however, were 
observed between overt and covert responding at the 
low information level or at the intermediate and high 
information levels when the responses were incidental. 

IMPLICATIONS OF THE STUDY 
This study supports Holland’s view that in an in- 
structional program responses should be relevant rather 
than incidental. It also suggests that the issue of overt 
versus covert responding can be resolved only as a 
function of the kind of material that is taught. It is 
entirely possible that with any given program certain 
responses should be made covertly and others, overtly, 
depending on the informational value of those respon- 
ses at the time. Thus, frames of a program could be 
flagged as to which ones should be responded to overt- 
ly. Replication of this study with other materials is 
deemed quite important. While the materials in this 
study were not what might be described as “meaningful 
materials,” it is important to ascertain whether the re- 
sults will hold when meaningful, systematic bodies of 
content (as one might find in a school curriculum) are 
taught. 



FILM AS A PROGRAM MEDIUM: TECHNICAL AND ECONOMIC CONSIDERATIONS 

Gifford M. Mast 

GIfFord Mast analyzes the cost of film and a projector for individual programmed 
instruction presentation. The cost of individual hardware is currently high^ but would 
be reduced substantially if the machines were mass produced. Raw film stock turns 
out to be cheaper than the paper needed to present the same number of frames. 

Major advantages of film appear to be its economical and flexible use in the short 
run, and for programs using color and illustrations. 



The economics of the film as a programmed learning 
medium vs. paper, like almost all of the other problems 
in connection with programmed instruction, has no 
simple answer. Our calculations indicate that for infor- 
mation content, film can be a more economical medium 
than paper as well as more versatile. I don’t think that 
film will replace paper in the broad field of education, 
but I do believe that use of film has greater possibilities 
than most people suppose. I shall try to show you some 
of the reason why I think so. 

To have an economical film-type teaching machine, 
it might at first appear practical to use just a magnifier 
and, in fact, one well known company took that ap- 
proach. But the simple fact is that you cannot get 
more than 2X or 3X magnification with a two-eyed 
magnifier without going to the relatively complex optics 
of a bmocular microscope. With a one-eyed viewer 
you can use up to about 5X or 6X magnification, but 
who would think of doing programs with one eye! 
With only 2X or 3X magnification, I think it is safe to 
say that film is not attractive as a program medium 
because of cost. 

If you abandon the magnifier type of viewer, you are 
then faced with a need for a projection device. And it 
was our conclusion that a projector with a lOX to 15X 
magnification was very practical for two reasons: (1) 
While you can get enough light through a piece of film 
to enlarge it much more than that and still have enough 
light to read your copy on the screen, the smaller the 
hole through which you put the necessary quantity of 
light, the more intense the light will have to be. The 
more intense the light, the more of it is converted to 
heat in a given area of film. Soon you arc faced with 
the situation that you probably have experienced at 
some time with an 8mm or a 16mm projector: if the 
projector stops, the film bums to a crisp in a few 
seconds. The only way to prevent this is to have elabo- 
rate provisions for cooling the film; and (2) A low 
magnification does not require as costly lenses as high 
quality film work, nor as precise location of the film, 
but provides a good quality picture when viewed at nor- 
mal reading distance. 

Klaus and Lumsdaine have written a paper about the 
“economic realities” of teaching machines. It is their 
contention that a teaching machine has to be limited 



to the $12.50 to $25.00 bracket to be practical. When 
you compare what you require of a film-type teaching 
machine with other things on the market, you find you 
have something comparable to the typical $100 auto- 
matic slide projector you find in every camera store. 
One such projector we examined recently had an esti- 
mated $100,000 worth of tooling, and the manufacturer 
had produced over 250,000 such units in a year. There 
might be some question whether we are yet in that 
bracket of possible market for teaching machines, pro- 
grams being the problem and as costly as they are. 
Therefore, I am going to assume, for the sake of this 
presentation, $150 to 300 is the range in which you 
might reasonably expect to find a film-type teaching 
machine being produced in low quantity. 

All this talk about machines is because we can’t use 
a film program without a machine and the cost of the 
machine is a significant factor in the problem of film 
vs. paper. While it may be dijfficult to prove by fully 
controlled, statistically valid methods, it is our belief 
that the right machine has a major advantage over 
books. As Dr. Douglas Porter stated, “It provides a 
more compelling presentation of Programmed Instruc- 
tion than a book.” Kaufman liked the “fascination” of 
the teaching machine and the fact that “they discour- 
age the trainee from looking ahead, from going to 
places in a program other than where he should be.” 
George Skroblus says, “a machine can prevent the 
trainee from looking ahead or skipping some of the 
material without proving he understands it,” and states 
that, “for some adults, we’ve found the teaching ma- 
chines have more appeal than a text, programmed or 
otherwise.” Whatever you may say against machines is 
offset by the difficulty of trying to find your place in 
Dr. Skinner’s book once you’ve laid it down! 

Consider the cost per class hour per student as a unit 
to get a dollars and cents handle on things. The teacher 
we shall assume is, or should be, paid $9,000 per year. 
If you use an Iowa school year for an example, the 
teacher teaches about 900 hours and therefore, costs 
about $10 per class hour, not including any overhead. 
Let’s assume the Talmud sanctified 25 pupils per class, 
so the cost is 40 cents per class hour per pupil for 
the teacher. 

Let’s see what the cost of the machine turns out to 
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be, taking it as a $200 item, good for 5 years at an 
amortized cost of $40 per year. For safety, add in 
$10 per year for maintenance, making $50 per year 
the overall cost of the machine. If the machines are 
used as many hours as the teachers, for whom we 
assumed 900 actual teaching hours per year, the cost 
of the machine per hour will be 5.5 cents per student 
hour. Of course, a machine being a tireless mechanism 
might be used for more than 900 hours per year thus 
reducing that cost figure substantially. 

We have estimated a cost to the school of about 
one cent per frame for copies of programs on films in 
cartridges for use in our teaching machine, this figure 
to represent a relatively small quantity production and 
to include the typical cost of programming. With sub- 
stantial production we believe that this cost could go 
down possibly 50%. Assuming 100 frames per class 
hour, which we believe to be high, our student would 
be using 50 cents worth of film each class hour. But 
bear in mind this film is tested for 1,000 uses, and a 
super-conservative figure should be 100 uses, so the 
cost of providing film is down to 0.5 cents per student 
per class hour. 

In a typical public school system, the cost of per- 
sonnel is about 85% of the annual budget. One way 
of saving money is to increase the size of the class. In 
fact, if you could obtain just as effective instruction in 
a class of 50 with programmed instruction on a teach- 
ing machine as you do with 25 and no machine, you 
would have about 20 cents per student class hour with 
which to play. 

The difference is demonstrated by aslcing: “How 
many students would we have to add to a class of 25 
to save the entire cost of teaching machines and filmed 
programs?”: (1) We have found the cost of a typical 
teacher is 40 cents per student with 25 students per 
class; (2) The cost of teaching machines and filmed 
programs we estimated at 6.0 cents per student hour; 
(3) Therefore, to offset the cost of using Programmed 
Instruction, the cost per student hour for the teacher 
must be reduced to 34 cents; and (4) Thus the number 
of students must be increased to 29: 

.40 

N = X 25 = 29 students 

.34 

In other words, with the parameters we chose, an 
increase of less than 20% in class size offsets the entire 
cost of teaching machines and filmed programs. 

Compare the cost of the programmed text with 
filmed programs. A hypothetical programmed text 
published in runs of about 50,000 copies consists of 
3,000 frames, is a paper back selling for $3.00, or one- 
tenth cent per frame. Publishers have told me that 
the expected use of a text is 2.5 students. Consequently 



the cost per class hour per student is four cents. This 
figure of four cents per class hour per student com- 
pares with the six cents per class hour using the ma- 
chine and filmed program. The former figure is based 
on a high production book, while the latter is on rela- 
tively low production, so it is reasonable to expect 
comparable production quantities to equalize these 
figures, or pass the advantage to film and machine. 

Let us examine the basic medium itself: The cost of 
raw 35mm film stock is 1.6 cents per foot. Each foot 
of film provides screen area of projection with our 
machine, using 9X magnification, of 24 sq. in. x 
32 = 776 sq. in. 

Each sheet of x 1 1 paper provides 92.5 square 
inches. Disregarding the fact that ample margins gen- 
erally must be provided on the paper but have already 
been deducted in our film calculation, we will say each 
sheet is equivalent to 4 frames, or 96 square inches 
on our screen. Paper (20# mimeo) costs in quantity 
about $1.10 per ream (500 sheets) or 0.22 cents per 
sheet. The equivalent raw film area is of a foot of 
35mm film costing 0.2 cents. Raw film costs less than 
raw paper. 

There is one particular area where film offers out- 
standing cost advantages. This is the area of the short- 
run program of which you may want 5 to 25 copies 
for testing and validation or 100 copies for industrial 
training, or even a few thousand copies of a national 
training program, or of a program at an advanced 
scholastic level. 

There the film eliminates: (1) Retyping on stencils 
or offset masters or typesetting, and (2) More impor- 
tant still, it eliminates the extra proof reading, which 
may cost more than the copying or typesetting itself. 
It makes possible the economical use of illustrations, 
whether they be line drawings, existing half-tones, or 
original photos. 

Digressing, in a sense, from strictly economic con- 
siderations, let us point out the great freedom in the 
use of illustrations that film provides the programmer, 
even during the testing and validation phase of his 
work. And if one picture is worth 1,000 words, we’re 
going to get a lot of “yakkity-yak” out of some pro- 
grams by using film as our medium! 

The possibility of using color, even in short run 
programs, is important. With a printed program, color 
plates can be an enormous cost per copy. With film, 
the raw print film costs 3.6 cents per foot instead of 
1.6 cents. Think what this can mean to sophisticated 
and therefore relatively short run programs for medi- 
cine, botany, zoology, physiology, crystal structure, and 
art. The raw material cost for 10 hours (800 frames) 
of full color program print is less than 60 cents more 
than for the same amount of program in black and 
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white. This added material cost provides, not just an 
occasional illustration in color, but the possibility of 
color in every frame and answer. 

Some one has said to us: “We aren’t ready to film 
these programs yet because they haven’t been fully 
validated.” He is wrong. One of the easiest places to 
effect an economy in program preparation is through 
taking advantage of what you can do with film and a 
projection type teaching machine. 

In summary, I think we can say that the economic 
side of using film as a programmed instruction medium 
is characterized by these points: (1) The machine is 
a substantial cost factor but will probably be small 



compared to the cost of programming itself except for 
high consumption programs; (2) The cost of raw film 
as compared to paper is more or less a standoff but 
can be expected to show a greater advantage for film 
as the technology develops; and (3) Major advantages 
of film appear at this time to be its economy especially 
in short runs, and its flexibility in permitting the use 
of color and illustrations. 

On the general side, we can say that film offers the 
programmer simplified mechanics and a freedom in 
the use of illustration and color. In fact, we are still 
looking for the program that really challenges the full 
versatility of film. 



differences IN MENTAl ALERTNESS AND 
IMPUCAflONS FOR THE SCHEDULING OF PROGRAMMED INSTRUaiON 

Wallace Bloom and John Pierce-Jones 

.1 *^®5°9nized that there are diurnal variations in vigilance 

that affect learning. Self-ratings on mental alertness showed substantial variations 
and, on a time continuum correlated positively with adjacent ratings and negatively 
with those further oway. The period from noon to 6:00 PM was most frequently 
reported as below average. Effectiveness of programmed instruction might be 
increased by scheduling it with due consideration of an individual's diurnal pattern. 
There is need for further research, particularly involving the scheduling of programmed 
instruction on an individual basis, to determine the full extent to which an individual’s 

diurnal variations may be a systematic rather than a random source of variance In 
human learning. 



By habitually observing socially determined sched- 
ules, specifying the hours for work, play, meals, and 
sleep, man has developed a dinural rhythm. Marsh 
(1906) reported that diurnal rh)?thm imposed a cor- 
responding periodic character on human efficiency. 
Kleitman (1949) found that efficiency of performance 
and degree of alertness seemed to conform to a rise and 
fall of body temperature. Pillsbury’s (1903) study of 
attention waves indicated there was fluctuation in their 
total length with time of day, and that attention waves 
corresponded in length with the Traub Hering wave of 
blood pressure. A twenty year study of workers in Mu- 
nich, Germany (Lehman, 1954), found variation in the 
amount of adrenalin in the subjects’ blood, the curve 
for which repeated itself even on non-working days. 
Renal excretory activity (Mills, Thomas, and Yates] 
1954), body temperature (Kleitman, et al., 1938; Carl- 
son and Johnson, 1937; DuBois, 1948; Murray, Wil- 
liams andLubin, 1958; Winslow and Herrington, 1949; 
MeUete, et al., 1951) water, chloride, and sodium ex- 
cretions (Lewis and Lobban, 1957) in subjects all 
shov/ed consistent diurnal variations. As stated by 
Pierach (1955), “No organ or organ system is exempt 
from the twenty-four hour rhythm in its function.” 
Thomson (1952) used three subjects to study the 
relationship of variation in skin temperature to general 
fatigue. He found that individual differences were pro- 
nounced in respect to temperature recordings and verbal 
reports of fatigue. 

Williams (1956) reported that body temperature, as 
determined orally, undergoes diurnal variations but, if a 
series of well young men are tested at the same time of 
day, considerable variation is observed. The tempera- 
tures of 276 medical students between 8:00 and 9:00 
A.M. varied from 96.6° to 99.4° F. He concluded that 
diurnal variations are related to sleep patterns which 
arc often distinctive, sometimes easily modified and 
sometimes not. 

From 1912 through 1931, Bjemer, Holm, and 
Swenson (1955) studied the errors made in entering 



figures related to gas production in ledgers of a large 
Swedish gas works in which 3 weekly rotating shifts 
were employed. The greatest number of errors occurred 
consistently at 3:00 A.M. regardless of which shift was 
on duty. Variations in errors were the same the last 
night of the week as on the first, so they concluded that 
one week was not enough to change the general pattern 
of the workers’ diurnal variations. There were neither 
winter vs. summer variations, nor 1915-1919 vs. 1920- 
1924 changes. Bjemer, Swenson (1953) found the 
highs on error curves were not due to physical fatigue 
and that three different groups failed to adjust to chang- 
ing shifts. 

Adams and Chiles (1961) subjected two B-52 com- 
bat crews to confinement for 15 days in a simulated 
space craft on an around-the-clock schedule of 4 hours 
duty and 2 hours rest. The diurnal variations for both 
performance and psychophysiological measures per- 
sisted throughout the 15 day period in spite of the sub- 
jects being removed from the usual day-night routine 
and environmental changes. 

Mackworth (1956) was concerned with the real psy- 
chological event that people failed to notice and act 
upon a si^al that they were quite able to detect. He 
conceived the measurable determinant of this psycho- 
logical event as VIGILANCE, “a state of readiness to 
detect and respond to certain specified small changes, 
occurring at random finip intervals in the external 
environment.” 

Deese (1955) stated tnat “largely due to the influ- 
ence of Mackworth the words ATTENTION and SET 
have been replaced by the term VIGILANCE'\ The 
concept of VIGILANCE conforms to the requirements 
of a concept in that it is “an abstraction from observed 
events ... a shorthand representation of a number of 
facts”. A subject may customarily have a low state of 
vigilance at a particular time of day, without it being 
caused by continuous work. 

Head (1923) contended that if the level of vigilance 
is lowered, specific mental aptitudes dies out as if the 
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voltage falls. “The more highly differentiated the act, 
the greater the degree of vigilance does it require.” 
Oswald (1962) applied this to the cerebral cortex and 
said “ . . . That the higher, within limits, the level of 
cortical vigilance in wakefulness (the level being de- 
pendent on the constantly fluctuating ascending facilita- 
tion from the reticular formation) the more complex 
data that can be dealt with.” He believed that our 
future understanding of the nature of attention will 
show it to be closely related to those mechanisms gov- 
erning cerebral vigilance which are manifest in sleeping 
and wakefulness. 

It is appropriate to give brief consideration to the 
positions of learning theorists regarding vigilance and 
diurnal variation. Thorndike’s (1913) Law of Readi- 
ness makes concession to the existence of the organism 
and variation in its state of vigilance. Hull’s systematic 
behavior theory contains several variables that are 
functionally related to diurnal variations in vigilance. 
In Hull’s own words: 

XVII the “constant” numerical values appearing in equa- 
tions representing primary molar behavioral laws vary from 
species to species, from individual to individual, and from 
some physiological states to others in the same individuals 
at different times. All quite apart from the factor of be- 
havioral oscillation (sor). (Hull, 1951). 

Freud assigned anxiety production to the ego and 
the normal habitual levels could be considered as vigi- 
lance. Taylor used this concept to create an anxiety 
scale, and found that the more anxious subjects were 
conditioned more readily than less anxious ones 
(Hilgard, 1956). This also holds true of vigilance 
when within optimal limits. 

According to Skinner (1957) cyclic changes during 
the waking hours, which may or may not be correlated 
with ingestion or other scheduled activities of the indi- 
vidual, are seen in both verbal and non-verbal behavior. 
McGeoch (1942) considered diurnal variation of vigi- 
lance as a source of uncontrolled residual variability. 
Hebb (1961) fully supported the existence of diurnal 
variations of vigilance and used the terms, “cyclical 
changes in excitability.” 

This investigation concerned college undergraduates 
utilizing programmed instruction materials. The specific 
purpose of the study was to describe the students pat- 
terns of self-reported diurnal fluctuations of mental 
alertness and efficiency. 

The subjects were 201 University of Texas under- 
graduates (male and female) taking a course in the 
psychology of adolescence by means of programmed 
instruction. Each subject recorded his evaluation of 
his own mental alertness and efficiency for each of the 
seven 2-hour periods starting at 8:00 A.M. as: (1) 
much below my usual level of performance, (2) a 
little below my usual level, (3) my average level, (4) 



a little better than my average, and (5) my peak per- 
formance. (The rating form is shown as Figure 1.) 

Name — 5 

Family First 

Course & 

Section: Code No: 

Self-Rating of Level of Mental Alertness and 
Efficiency at Various Times 

T his form is to record your own evaluation of your 
own mental alertness and efficiency at various times 
of the day. Consider how well you believe you study, 
organize and perform academic activities. Make ONE 
check mark for each two hour period. 

Rating Scale 

1. Much below my usual level of performance. (I 
find it hard to get started to concentrate; or get results 
in the usual period of time.) 

2. A little below my usual level. (I must make an 
extra effort to keep my attention focused or it takes 
a little longer than usud to perform the task.) 

3. Function at about my average level. 

4. Doing a little better than my average. (Easier 
to keep alert in class and accomplish assignments.) 

5. My peak performance. (Work is performed most 
easily and quickly.) 

Note: More than one period may be checked for each 
of the above ratings but each of the five should be 
checked somewhere in the time period. 

The results of the study are shown in Tables 1 and 2. 
Table 1 shows that comparatively few individuals con- 
sidered themselves as performing above their average 
during the hours of noon to 6:00 P.M. This, along 
with the evidence that more than half believed they 
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Table 1 



College Undergraduates, Self Appraisal of Mental 
Alertness and EfRciency at Different Times of the Day: 
Spring 1962 



Lowest 

Time 


1 


Average 
2 3 4 


5 


Highest 
X 0 


8-10 A.M. 


46 


29 


30 


41 


55 


3.15 


1.53 


10-12 N. 


2 


29 


39 


57 


74 


3.66 


1.10 


12-2 P.M. 


51 


43 


71 


29 


7 


2.49 


1.12 


2-4 P.M. 


30 


59 


68 


31 


13 


2.69 


1.10 


4-6 P.M. 


57 


60 


48 


25 


11 


2.37 


1.18 


6-8 P.M. 


39 


42 


44 


44 


32 


2.94 


1.36 


8-10 P.M. 


22 


17 


34 


48 


60 


3.73 


1.35 



N-261 



were below their average during this same period, ap- 
pears particularly relevant to the scheduling of pro- 
grammed instruction. Within the range of administra- 
tive restrictions, the uniqueness of individual patterns, 
such as the “Early Bird” and the “Slow Starter” whose 
diurnal patterns are shown in Figure 2, probably should 
be taken into account when preparing individual sched- 
ules. Of course, such a suggestion as this takes it for 
granted that Ss rate their efficiency with better than 
chance validity and that learning and performance 
variance is reliably associated with such self-ratings 
by Ss. In this connection, it can be noted that Welford, 
Brown, and Gabb (1950) found a relationship between 



self-rated fatigue and complex task performance, and 
Fraser and Samuel (1956) found a significant associa- 
tion between self-rated fatigue and vigilance. Shaw’s 
(1954) work with air traffic controllers did not, how- 
ever, produce confirming results. 

Subjects’ ratings for each period were correlated 
with their ratings for each of the other periods. The 
results are shown in Table 2. In general, the findings 
indicated that the self-appraisal ratings at a specified 
period were positively correlated with the adjacent 
periods but negatively with all other periods. This 
confirmed the existence of a systematic rhythm rather 
than a random pattern. 

Examination of individual reports showed, as would 
be expected, many typical patterns similar to the aver- 
age trends. Predictably, of course, a few “early birds” 
started the day at a high level and decreased con- 
tinually after noon, and a few “late starters” began 
the day at a low level, reached their average mental 
alertness in the afternoon, and were apparently highest 
in the evening. (See Table 2.) This is consistent with 
the findings of Freeman and Hevland (1934) who 
reviewed 69 research reports on diurnal variations in 
performance, found there was no single pattern, and 
classified the patterns into four general categories. In 
34 reports on mental performance, they found: 12 
Type I, continuous rise in performance; 12 Type II, 
continuous fall in performance; 5 Type III, morning 
rise, afternoon fall; and 5 Type IV, morning faU, after- 
noon rise. These four types discovered in their review 



Table 2 

Correlation of Changes in Seif Appraisal Ratings of Mental Alertness 
and Efficiency at Different Times of the Day 





8-lOAM 10-12N 


12-2PM 


2-4PM 


4-6PM 


6-8PM 


8-lOPM 


8-10 A.M. 


.1787* 


-.1766* 


-.2924** 


-.3413** 


-.2767** 


-.3957** 


10-12 N. 




.0820 


-.1563* 


-.2327** 


-.1728* 


-.2376** 


12-2 P.M. 






.1759* 


-.0887 


-.2591** 


-.2088** 


2-4 P.M. 








-.1883** 


-.1793* 


-.1969* 


4-6 P.M. 










.0607 


.0531 


6-8 P.M. 












.2689** 



8-10 P.M. 



Note. N = 201 

*p = .05 **p = .or 
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by Freeman and Hevland, may have been due in part 
to the selection of subjects from one socially determiaed 
schedule by some researchers and of subjects from 
other schedules by other researchers. 

The probable importance of scheduling for pro- 
grammed instruction in relation to an individual’s 
diurnal pattern can be illustrated by an actual example. 
One male subject reported that after he had filled out 
the self-rating form, he had thought further about his 
own daily variations in mental alertness. His hours in 
the programmed instruction laboratory were self- 



selected and he customarily came in at 1:00 P.M. and 
studied for an hour. Invariably he had not completed 
a lesson (approximately 72 jhames) in his hour and 
had to file it in the “pending drawef ’ and complete the 
last frames at his next laboratory period. It usually 
took him six periods to complete five lessons. One 
morning when he made up a missed period, however, 
he completed the entire lesson. Thereafter he attended 
the laboratory in the morning, finished a lesson each 
hour, and often had time for a cigarette or soft drink 
before his next class. 



WHAT SHOULD BE PROGRAMMED FOR TELEVISION? 

George L. Cropper 

George Gropper analyzes the capability of the television medium to meet the 
stimulus-response requirements of various subject matters. He concludes that television 
is uniquely qualified for certain types of visual presentations and that optimum use 
will be made of the medium if restricted to such presentations. 



During the past few years, programming techniques 
have been adapted for use in instructional television. 
1961) in which, among other more analytic findings, 
Arthur Lumsdaine and I have reported on U. S. Office 
of Education Title VII research (Cropper & Lumsdaine, 
students who responded to programmed sequences 
out-performed those who merely viewed conventional 
lessons. At State Park, Pennsylvania, Carpenter and 
Greenhill are completing research in which program- 
ming techniques have also been adapted to television. 
In studies currently in progress (Gropper, 1962) I have 
been engaged in applying programming techniques 
to the presentation of science demonstrations. Does it 
make sense to apply to group instruction programming 
techniques and methods developed for individual in- 
struction and if so, how? 

If by programming we merely mean exercising ex- 
plicit control over student learning behavior, the ques- 
tion that should be asked is not “should TV be pro- 
grammed?” but rather, “what should be programmed 
for TV.” There can be little question about the desir- 
ability of controlling the learning process as closely 
and as explicitly as possible, whether students are learn- 
ing from teachers or teaching machines or texts or 
even television. There can be some question as to 
whether some subject matters can more effectively 
be taught by means of one medium rather than by 
means of another. I would like to suggest some guide- 
lines for deciding what are appropriate types of subject 
matters for presentation over television. 

We have all in varying degrees paid some measure 
of lip service to the systems approach to the problems 
of training and education. We have tended to say, let 
us choose each medium for the job it can do best, 
whether it is a teacher, a programmed text, or a teach- 
ing machine. But it has not been easy to determine ex- 
actly what television can do best. I would like to suggest 
two well known criteria which will help us make this 
decision: cost and achievement. Under cost we may 
include such things as time, administrative convenience, 
and human and physical resources. Achievement is 
the end product of the efforts of both those being in- 
structed and those providing the instruction. In terms 
of both criteria the functional capabilities of instruc- 
tional television can be assessed. 



Cost 

Instructional television is an extremely expensive 
medium. The costs of planning, preparing, producing, 
transcribing, and transmitting lessons are quite high. 
If we follow current practice of not accepting lesson 
sequences until after pre-testing and revising so that 
they might reach some prescribed standard, the costs 
of televised instruction can conceivably double, triple, 
or multiply even more. But the ink in the ledger is 
not all red. The traditional argument for TV is that 
many students can be reached at the same time. When 
costs are prorated over many students, per student 
costs may in fact be quite low. The critical question 
thus becomes whether the many students reached by 
television are learning as effectively as possible. 

Achievement 

One of the major problems encountered in Educa- 
tional Television is the problem of individual difference 
or student capability. The results of studies which we 
have done at the American Institute for Research, 
working jointly with Metropolitan Pittsburgh Educa- 
tional Television Station, WQED, in Pittsburgh, Penn- 
sylvania, have shown that there are difficulties in ac- 
commodating individual differences when providing 
instruction for heterogeneous groups. One attack on 
this problem, which we are currently pursuing, is to 
explore the possibilities of creating homogeneous 
groupings based on learning rates and of adapting in- 
struction suitable for these groupings. It is hypothesized 
that in this way higher levels of achievement will be 
realized. 

What are the characteristics of TV, and of other 
media, which influence the effectiveness of instruction, 
and what are the characteristics of subject matter con- 
tent which interact with characteristics of the instruc- 
tional process? Subject matter areas differ, of course, 
with respect to the types of stimulus materials which 
are integral to them. Physics, for example, involves 
both visual and verbal stimulus materials. History, 
on the other hand, can more usually be taught with 
purely verbal materials. The visual presentations in 
physics are likely to be both static and dynamic. Visual 
presentations in botany, on the other hand, are more 
apt to be purely static. Similar differences occur in non- 
academic training areas. Visual tracking of moving 
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objects involves transient stimuli. The assembly or 
disassembly of objects typically involves stimuli which 
are static and also dynamic. 

Media differ in both their display and response capa- 
bilities. They vary in the complexity of their capa- 
bilities for displaying stimulus materials. Some me- 
dia, such as film or TV, have a highly complex 
capability of displaying both visual and auditory stim- 
uli of great range and variety. Other media, such as a 
textbook, have much less flexible and much less varied 
capabilities for presenting the range of relevant stim- 
uli required by diverse subject matters. Media can 
be ordered in terms of the complexity of tliese capabil- 
ities for displaying relevant subject matter stimuli. The 
medium which is used to create the learning opportun- 
ity thus must have both the appropriate display and 
response capability, so tliat students or trainees be 
permitted to practice relevant reponses in their pres- 
ence. If the relevant stimuli are transient, the medium 
should be capable of displaying them, e.g. films or 
TV. If the stimuli are static, the medium should be 
similarly capable of displaying them which is a much 
less difficult requirement to meet. 

Media can also be ordered in terms of their capabil- 
ities for controlling the instructional process, such as 
the ability to control the amount, timing, and duration 
of displayed stimulus materials. Film and TV, for ex- 
ample, differ from teaching machines or textbooks 
with respect to the rate at which material is presented 
and with respect to who controls the rate. These fac- 
tors related to the explicit control over stimulus display 
have a significant bearing on achievement to the ex- 
tent that they facilitate or interfere with the prac- 
tice of correct responses. 

In the case of television, there is a highly flexible 
visual and auditory capability. Precise control over 
amount and type of material is possible. It is possible 
to arrange conditions such that responses are 
prompted only in the presence of specified stimuli. 
These conditions favor the practice of correct respons- 
es; others do not. The fixed-pace character of films 
and television can interfere with the capacity of the 
stimulus materials to cue correct responses. There is 
no self-pacing capability which might interact with the 
specially prepared stimulus materials in allowing and 
indeed encouraging correct responding. The same 
specially prepared stimulus which can successfully cue 
correct responding when the student is allowed as much 
time as he needs, may be inadequate for the same 
purpose if an external time limit is imposed on the 
student. It is here that the problem of individual dif- 
ferences becomes serious. 

It is entirely possible to alter stimulus materials or to 
allow sufficient time, determined empirically, so that 



few or no students will be left behind; but the itei 
the heterogeneity of the group, the more likely it is that 
this lowest common denominator will have its inherent 
drawbacks. Other approaches are possible. Thus, not 
only w'ill time be varied, but also the content. For ex- 
ample, strength of cueing or amount of practice or 
review might differ for fast and slow learners. Tliis 
is an approach we are currently investigating, using 
learning time which is characteristic of individuals as 
a basis for sorting them into homogeneous groups. Of 
course, when one is dealing with a relatively homo- 
geneous group to begin with and when in addition the 
group is a high-ability level group, say a college popu- 
lation, a single grouping may be quite sufficient. The 
median time required may prove successful as in re- 
ports from research at Pennsylvania State University. 

It was pointed out earlier, in connection v/ith sub- 
ject matter content, that media can be ordered with 
respect to the complexity of their capabilities for pre- 
senting stimuli relevant to the subject matter, rele- 
vant stimuli to which students may be encouraged to 
practice relevant responses. Here, too, in connec- 
tion with the instructional process, we can order media 
in terms of their capacity to present stimuli in such 
a way as to facilitate correct responding. Television 
was ranked quite high with respect to its capacity' to 
display relevant stimuli to which relevant responses 
might be made. In the case of the instructional proc- 
ess, because of its fixed-pace character, it may have 
to be ranked relatively low with respect to its capacity 
to prompt correct responding. A third consideration or 
criterion for evaluating media is the logistical or 
economic one. Thus media may be ranked in terms of 
three criteria: (a) capability of displaying stimuli; 

(b) capability of controlling condition of display; and 

(c) cost. 

In deciding which medium to use for a particular 
subject matter, it would appear that one ought to start 
with the criterion having to do with the capacity of a 
medium to display relevant stimuli to which appropri- 
ate responses can be made. A pragmatic approach 
would call for the selection of the medium with the 
minimum capability to do the job. The minimum is 
suggested because it would leave more complex media 
available for more important jobs, and it would also 
tend to be the most economical approach, since com- 
plexity of capability and cost are likely to be directly 
related. The second consideration might be the ca- 
pacity of the medium to facilitate correct responding. 
When on the basis of the first criterion more than one 
medium might do the job, media can then be judged 
in terms of the second criterion, namely control over 
the instructional process. Over and above correct re- 
sponding, other control features such as response 
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provisions, safeguards against such things as cheating, 
etc., would be considered. Cost considerations, the third 
criterion, would enter only after these achievement 
considerations had been settled. Judgments about 
media found equally capable in terms of the first two 
achievement criteria would be based finally on money 
considerations. 

What I have been describing, of course, is the opti- 
mum way of making decisions about media utilization. 
In any practical situation, all three sets of criteria we 
have been discussing are likely to be considered jointly. 
Compromises and trade-offs are likely to be found nec- 
essary for a variety of reasons. These might include: 
the desire to make full use of media already paid for 
and available rather than acquiring new media facili- 
ties; the administrative convenience made possible 
through the use of one medium rather than another. 
Compromises m media selection may result in under- or 
over-utilization of a medium. A medium is under-util- 
ized when it is used to display stimuli which a medium 
of less complex capabilities can handle. It is capable 
of a far more complex job. For example, television is 
under-utilized when it is used to present printed mate- 
rial which could be presented equally well or better in 
a textbook, teaching machine, or on slides. If avail- 
able and paid for, and if there is the desire to make 
fuller use of the more complex medium, it is possible 
to use the more-complex-than-necessary medium for the 
simpler job. It may be used, however, only so long 
as it brings with it no liabilities. In the case of televi- 
sion, while it is fully capable of displaying printed ma- 
terials, it nevertheless imposes a number of individual 
differences problems. To the extent that these cannot 
be offset, the medium would not only be under-utilized, 
it would also bring with it serious liabilities. When a 
medium brings with it no such liabilities, its capabil- 
ities may be under-utilized but they may still provide 
effective instruction. 

Over-utilization arises when a medium of simple 
capabilities is expected to do a job requiring far more 
complex capabilities. The use of a succession of slides 
to try to simulate motion or transient stimuli, which 
can only be effectively displayed by means of film or 
TV, is an example of over-utilization. In contrast to 
under-utilization (which brings with it no liabilities), 
over-utilization appears to be the far more serious 
problem to the educator because it is likely to be more 
costly, in terms of unacceptable standards of achieve- 
ment, than are compromises which err on the side of 
under-utilization. 

Using TV as a kind of group teaching machine for 
presenting strictly verbal material would appear to be 
a waste of the medium’s far more complex capabilities. 
It would constitute, it seems to me, under-utilization. 



George L Gropper 

It may also not be adequate to the job of accommodat- 
ing individual differences, at least not at an economical 
cost. The creation of homogeneous sub-groupings and 
separate presentations for them will increase per- 
student costs. But TV may be required to do an ef- 
fective job in presenting dynamic or transient stimuli 
in a way that less complex media cannot. It can pre- 
sent science demonstrations, either to teach facts and 
concepts or to teach laboratory procedures, which 
would not adequately be handled by an individual 
teaching machine of far less complex capability. Televi- 
sion would appear to be suitable for training involving 
visual discriminations aboui transient stimuli (e.g. radar 
scopes, identification of moving objects, landing a 
plane, and steering vehicles). The nature of the sub- 
ject matter stimuli determines what is the most ap- 
propriate medium. 

If, as suggested earlier, what we mean by program- 
ming is the control over learning behavior, then the 
“what” of “what should be programmed for televi- 
sion” are visual presentations. They are the visual pre- 
sentations which are required by the subject matter 
and which cannot be effectively displayed by other 
media. 

While it appears highly desirable that responses to 
visual presentations be made under the explicit control 
of specified parts of a presentation and that they be 
made actively, as they often are to verbal stimulus ma- 
terials, this has not been typically the case in filmed 
or televised instruction. It is feasible to attempt to 
require active responding to visual stimuli whether the 
responses consist of visual discriminations, verbal re- 
sponses, or non-verbal motor responses. While re- 
sponses other than those being controlled may also be 
made, which are usually referred to as implicit re- 
sponses, to the extent that stimuli and responses can 
be specified, greater control wiU be exercised over the 
learning process. 

In the experiment mentioned at the beginning of this 
paper (Gropper, 1962), an entirely visual lesson on 
Archimedes’ Law was programmed for television pres- 
entation. The students were encouraged to make visual 
discriminations about the events of a demonstration 
illustrating the concepts covered by Archimedes’ Law. 
Students made discriminations about scale readings for 
objects weighed in and out of water; about amounts of 
water displaced by objects weighed in and out of water; 
about amounts of water displaced by objects of differ- 
ing size; about the magnitude of buoyant force for ob- 
jects displacing varying amounts of water; about the 
magnitude of the apparent loss of weight for objects of 
varying size; and finally, about the inter-relationships 
among all these phenomena. 
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The demonstration was broken down into small steps, 
sequenced, and organized in such a way as to prepare 
students to make the relevant discriminations correctly. 
Students actively made the discriminations by select- 
ing the correct answer from appropriate choices. The 
demonstration was pre-tested and revised until it was 
relatively error free. On the basis of this lesson stu- 
dents were ultimately able to answer both verbal and 
pictorial achievement test items. Equally as important 
as demonstrating the possibility of teaching essentially 
verbal material by non-verbal visual means is the de- 
monstration that it is possible to control student re- 
sponses during visual presentations. The more pre- 
cise and extensive this control is the less likely are 
adventitious factors to intrude. 

Static or dynamic visuals may serve a number of 
functions in the control of student responses. These 
have been described elsewhere in detail (Cropper, 
1963) and will be described only briefly here. Visual 
displays may be used to cue responses in any of the 
modes we have described earlier. In this capacity vis- 
uals which already control such responses are likely to 
be selected. Similarly, such visuals may also be used 
to provide confirmation for responses, i.e., responses 
predicting events or responses indicating which of sev- 
eral static visuals is the one appropriate to a given 
occasion or situation. When the visual is one of a class 
of visuals to which the student is expected to respond, 
it may also serve as an example used to facilitate the 
acquisition of a generalized response. Understanding 
of a principle is thus built up through the use of a series 
of visual examples. 

In serving a cue, reinforcement, or example function, 
visuals rather than words are used for one of two rea- 
sons. A visual may be relevant to and an integral part 
of the subject matter. It therefore appears in the cri- 
terion situation and the student is expected to respond 
appropriately to it. Experiments in science are a good 
example. TTie student, following instruction, is often 
expected to observe, describe, or interpret physical 
events. The visually perceived events are criterion 
events. Visuals used in instruction may be identical to 
or highly similar to those found in the criterion situa- 
tions. These visuals may be called criterion visuals. 
The object of instruction is to bring appropriate re- 
sponses under their control. 

Other visuals can be used in the learning situation 
despite the fact that tliey do not occur in the criterion 
situation. They are used primarily because the re- 
sponses that we wish the student to practice are already 
under their control. The visuals are therefore used to 
cue or prompt these responses so they may be brought 
under the control of other stimuli, usually veibal stim- 
uli. For example, graphs and charts can be used to cue 



responses which may then be brought under the control 
of other stimuli. The graphs or charts are not typ- 
ically found in the criterion situations. Such visuals 
may be called intermediary visuals, denoting their 
instrumental and temporary use. 

Intermediary stimuli appear to hold promise as being 
the visual counterpart of familiar verbal stimuli already 
in the student’s repertory. We use familiar words, 
ideas, or concepts to prompt responses which prepare 
the student to learn new responses. Students can al- 
ready respond meaningfully to many properties of vis- 
ual materials. We can, therefore, use such visually 
perceived stimuli as lines, boxes, arrows, graphs, etc., 
to prompt responses, just as familiar words prompt re- 
sponses, so that their control can be transferred to 
other stimuli Familiar visual materials may thus be 
added to familiar words as further tools in shaping 
the acquisition of new behavior. 

Both film and TV possess highly rich and flexible 
visual capabilities, not to mention audio capabilities. 
These capabilities can be used instrumentally to 
control the conditions under which learning takes place. 
This might include the control over the amount and 
timing of the presentation of stimulus materials; and 
control over the opportunities for responding in the 
presence of particular stimuli and not others. These 
same capabilities may also be employed to exercise 
control over the content presented. They can be used 
to present stimuli relevant to a subject matter which 
other media cannot present. They can be used to pre- 
sent criterion visual stimuli and intermediary visual 
stimuli which are to control the practice of responses 
which, as dictated by instructional objectives, are rele- 
vant and correct. If we seek to use these visual mate- 
rials presented by film or TV in as systematic a fashion 
as we have verbal material presented by means of other 
media, we stand a good chance of finding a cornucopia 
of valuable tools, as yet minimally taken advantage of, 
to be used in our new and expanding instructional 
technology. 

Television ought to be used for those subject matters 
where relevant stimuli are visual and are dynamic or 
transient. We accept some of the liabilities of the medi- 
um imposed by its fixed-pace character. For the pur- 
poses of accommodating individual differences, we may 
settle for an instructional sequence that is too long for 
the fast learners but which is needed by the slow learn- 
ers. In terms of time we have not been as efficient as 
possible, but in terms of achievement we are likely to 
be highly efficient. We reach this kind of achievement 
efficiency because of television’s capability for display- 
ing relevant visual stimuli. Used in this way, television 
is the visual medium it has so often been described to 
be. 



PROBLEMS AND ISSUES IN THE PUBLISHING OF PROGRAMMED MATERIALS 

John R. Everett 

John Everett sfofes that a pubiiaher li clearly at fault if o teacher says one of his 
programs fails fo teach. It is the publisher's {ob to insure the quality of his product 
by conducting extensive field tests. The testing and consequent revisions are quite 
costly, and will raise the price of a program above that of a textbook. But the in- 
creased cost is well worth the increased effectiveness of the product. 



As far as we know, the kind of printing that we have 
known in the West had its origins in the middle of the 
15th century in Mainz, Germany. Since that time West- 
ern man has enjoyed the enormous benefits of ideas 
and facts flowing from mind to mind by way of the 
printed page. It is fitting that those interested in the 
flow of ideas and facts recognize that mass production 
started with Gutenberg and not with Henry Ford and 
that we attempted efficiently to satisfy the mind lo ng 
before we applied the efficiency of manufacturing inter- 
changeable parts to the task of satisfying the desire 
for movement. 

From the time of Gutenberg to just a few years ago 
the format for knowledge presentation in book form 
changed very little. A glance at one of the illummated 
pages from the Gutenberg Bible indicates that the 
words are strung together in sentences, groups of sen- 
tences are divided into paragraphs, illustrations are 
scattered around the page, and color is lavishly used 
in the illustrations. There have been many changes in 
the technique of printing since 1454, but most of the 
books used today still divide words into sentences, 
sentences into paragraphs, paragraphs into pages, and, 
in many cases, color the illustrations. 

The way in which knowledge was presented on the 
first printed pages has an ancient history, stretching 
far back into the years before movable type was in- 
vented. The author hoped he used words that conveyed 
his meanings and his craftmanship was judged by his 
skill in using grammatical conventions and his ability 
to select the right word. Since our earliest written al- 
phabet language, perhaps between 1000 and 600 B.C., 
our grammars have been refined and our dictionaries 
have given increasingly precise meanings. But the basic 
form of written communication has remained the same, 
with an author hoping that the meanings he intends are 
the meanings his readers receive. 

The purpose of this excursion into history is to indi- 
cate just how^ radical is the change being created by 
programmed material. The printing press is the same, 
but the organizational pattern of the page is truly dif- 
ferent; the words are in sentences, but rarely are they 
in paragraphs; the lonely author writing in the hope 
that his words mean what he thinks they mean is re- 
placed by a group of writers constantly testing their 
meanings on groups of their readers. 



Now, how does this affect pubh'shing? 

In the past the task of a publisher was to find an 
authoritative author, give him an advance (small as 
possible), wait for a manuscript, hand it to an editor 
(as underpaid as possible), let the editor clean iQ) the 
grammar and the organization, send the copy to the 
printer, proofread it, and shower as many copies on 
the market as his sales force thought they could sell. 
The publisher stood between the author and printer, 
and the printer and the market. Very often he has 
viewed himself as the amoral midwife who cares not 
how the child is conceived, but whose only task it is 
to see that the baby lives. 

The publication of programmed materials requires 
that a publisher deal with not one author, but with a 
whole host of experts such as programmers, subject 
matter people and test subjects before he ever sees a 
manuscript. He must satisfy himself that the subject 
matter people know their business and he must know 
that the testing procedure keeps the programmers in 
line. And even after he has done all of this, he stUl 
is not sure when he gets the completed manuscript that 
it is a valid teaching instrument for 5,000 school chil- 
dren. The publisher must go back again to his cus- 
tomers and test again on a large sample of school chil- 
dren if it is text material. 

The standard traditional text book always had two 
other groups, the teachers and students, ready to ac- 
cept blame for non-learning. Both the publishers and 
the authors would loudly proclaim that they had an ex- 
cellent book, but the teachers misused it and the poor 
students suffered. Since the teachers couldn’t accept 
such blame without some kind of answer they would 
counter with the statement that the book was fine, but 
not written for their special students. And if all else 
failed, everyone would turn on the student and say that 
he obviously didn’t work hard enough and that the 
modem generation didn’t have any motivation any- 
way. 

The underlying assumption of programmed instruc- 
tional materials is that if the student doesn’t learn the 
fault lies directly with the program and the teacher; 
and in many ways, more with the program than the 
teacher. If the subject matter is sound and the pro- 
gramming has been skillfully done, neither student nor 
teacher can be blamed for a lack of learning. The 
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publisher doesn’t even have a single author to blame. K 
the publisher has not had his program tested and val- 
idated before he releases it, there is only one final rest- 
ing place for the cause of the poor results. Program- 
med instructional materials have finally broken the 
happy circle which made it possible for adults to be 
guiltless when children failed to learn. And, even more, 
teachers can now look publishers squarely in the eye 
and say, “I care not for your illustrious board of editors 
and for the fine names you place on your covers, the 
stuff we bought just plain doesn’t work!” 

It is my guess that those who use programmed learn- 
ing materials are going to have to learn how to evalu- 
ate properly the claims of publishers regarding their 
testing procedures and results. As far as I know there 
isn’t a person alive who can sit down with a program 
and tell you within a few hours whether it is a good 
program or a bad one, and whether it will teach or not. 
There are countless subject matter people who can 
pick up a traditional textbook and in very short order 
give a reasonably sound judgment regarding the ade- 
quacy of its coverage, the elegance of its style, and the 
soundness of its interpretation. We have lived with the 
traditional format so long that we know what to look 
for and we know where the guide lines are and where 
they lead. 

The genius of a good program is that it is so con- 
structed as to teach certain things about the subject 
to someone who knows nothing about it. All an expert 
in the subject can do is to tell you that the coverage 
claimed is actually on the pages. He cannot tell you 
that the program will teach. Nor can he tell you whom 
it will teach. An experienced programmer can tell 
you what he thinks the program might teach, but he 
cannot tell you that it will teach. Only a fairly good 
sample of students who have actually taken the pro- 
gram and who have actually been tested for compre- 
hension can tell you if the program teaches. Both pub- 
lishers and the users of programmed materials must 
become much more sophisticated regarding the tech- 
niques of testing and the analysis of statistics. The 
days are over when a publisher’s editorial board can 
nod happy approval when someone says in low seri- 
ous tones that Professor Ho Ho has looked at this pro- 
gram and has pronounced it good. And the same day 
is also past for the curriculum area specialist and 
teachers. 

Programmed instructional materials probably should 
be reviev/ed in professional journals only by people 



who know how to evaluate testing procedures and test- 
ing results. Those who have always known that fast 
footwork with statistical data has been developed into 
a high art surely must recognize the engraved invita- 
tion to fraud that the spread of programmed instruc- 
tional materials has issued. Publishers and buyers 
must set up rigorous validating and statistical inter- 
pretative machinery or Gresham’s law will surely slow 
down a remarkably fruitful movement. 

The unfamiliarity of the format, the fact that pro- 
grams are produced by groups of people rather than by 
a single author, and the amount of expensive testing 
that is required all conspire to make programmed in- 
structional material more expensive than the traditional 
textbook. I see no way around this necessary increase 
in the cost of instructional materials. It is not that a 
publisher will make a larger profit from the increased 
price. Rather, it is that there are more people and longer 
time spans involved. It can cost as high as $2,000 to 
validate and test a program that has already cost $10 
per frame to create. If a program is 4,000 frames long, 
an honest publisher can have as much as $42,000 tied 
up before the program is even set in type and manufac- 
tured. Put this off against a book where the publisher 
gives a $2,000 advance and has no further costs imtil 
the manuscript is in the house and ready for the print- 
er and you begin to see the magnitude of the financial 
problem. 

Schools, industry, and government are simply going 
to have to learn to live with the higher costs. Gi course, 
this shouldn’t be difficult because a good program 
comes to them with virtually all of the guesswork taken 
out and with a kind of warranty that has never before 
been possible. It is worth paying more for a car that 
works 99% of the time than would be paid for one 
that works 10% of the time. 

While any revolution is in progress there are always 
people on the outskirts of town looting the stores and 
stealing the gold. The revolution in instructional tech- 
niques which programmed materials represent will 
and has attracted its share of despoilers. Publishers, 
perhaps more than any other single group, must bear 
the responsibility for limiting such activity. It can be 
done if publishers temper their desire for quick profit 
with a serious concern for the long range future of their 
businesses and the educational health of their nation. 
They cannot be amoral midwives but rather they must 
keep alive only the babies who are properly conceived. 
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PROBLEMS IN PUBLISHING PROGRAMMED MATERIALS 

C. J. Donnelly 

C J. Donnelly points out the complex problems and decisions that must be made 
before one can begin large-scale publishing of programmed instructional materials. 
He suggests that traditional concepts of textbook publishing are being changed — 
beneficially and irreversibly — by virtue of the learner-oriented pre-testing that now 
precedes publication of programmed materials. 



The first two major producers of programmed in- 
struction materials were reference book publishers as 
distinct from textbook publishers. This paper repre- 
sents the thoughts of one of these two organizations, the 
TMI-Grolier affiliation, sharing some of its experiences 
in the publication of programmed materals. As one of 
the first large-scale publishers, we probably have made 
one of every kind of publishing decision posed in this 
field, and have had the opportunity, at least, to make 
one of every kind of mistake. Perhaps a discussion of 
our mistakes and non-mistakes would be instructive to 
those now entering the field. 

As Grolier approaches its fourth year of operation, 
we sense a general maturity overtaking the profession. 
No longer are words like “astounding,” “revolution- 
ary,” and “incredible” in the vocabulary of the mar- 
keting and publicity functionaries. One can discern an 
inverse relationship between the ratio of adjectives to 
nouns in advertising copy and recency of entry into this 
field; the more recent the entry, the more adjectives. 

At the moment, Britannica and TMI-Grolier still 
account for the majority of programs in print and com- 
mercially available. For those who fancy end-to-end 
statistics, these companies together account for some 
138,000 of the 193,000 frames now in print or 1,870 
of 2,478 hours of instruction. The Center for Pro- 
grammed Instruction points out, though, that this pic- 
ture will change rather significantly this year. For 
example, twenty textbook publishers have announced 
plans to publish a total of forty or fifty programs be- 
fore the end of 1963. There will be about 350 programs 
in the U.S. Office of Education-sponsored “Programs 
’63“ catalog, replacing “Programs ’62,” which listed 
about 100 programs. 

Now a few words about the role of the publisher, as 
I see it. It will be convenient to discuss this in two 
major subdivisions: (1) the publisher as interpreter 
to the general public, and (2) the publisher as a 
businessman. 

As interpreter to the general public (and here I in- 
clude the vast majority of practicing schoolteachers) 
the publisher necessarily and inevitably is cast in the 
role of marketing agent. While various organizations, 
including the U.S. Office of Education, the Center for 
Programmed Instruction, the National Education Asso- 
ciation, the National Society for Programmed Instruc- 



tion, the American Management Association, the 
American Psychological Association, and the Ameri- 
can Society of Training Directors have most effectively 
disseminated and promulgated research results, pro- 
gram and machine surveys, and general ioformation 
in the field, nevertheless, the Educational Sales Divi- 
sion of TMI-Grolier continues to report many unan- 
swered questions at the grass roots level. Like it or 
r.ot, it would seem that the majority of what teachers 
vtU learn about programmed instruction in 1963 will 
be learned from representatives of the educational pub- 
jjshing industry. A majority of the programs which will 
be used in the 1963-64 school year will be selected 
largely on the basis of this information. The publishing 
fraternity therefore welcomes all possible guidance from 
institutes and conferences on programmed instruction, 
and from the guidelines of such organizations as the 
joint APA-AERA-DAVI Committee. 

A few words about the business-end of the publish- 
ing process in the programmed learning field should be 
instructive. It is here that the myriad operational de- 
cisions which may appear minor, or even trivial, are 
made and it is here that the difference between opera- 
ting at a profit or a loss is determined. (And if it 
turns out to be a loss, the publisher cannot for long 
function in his other role as interpreter and “mission- 
ary.”) Even were the publisher not underwriting the 
considerable costs of program development, through 
advance royalty arrangements, long-term sole-source 
contracts and the like, a good bit of the shareholders’ 
money will still be committed on a risk basis every 
time a publishing decision is made, and a print order 
issued. TMI-Grolier alone, in the printing of programs, 
used 1,000 tons of paper in 1962. (Which is one of the 
reasons we are actively exploring film techniques.) 
We also accounted for something like 500 tons of plastic 
for the relatively uncomplicated, manually-operated, 
auto-instructional device we call the MIN/MAX. 

Once the basic decision to enter the programmed 
learning field has been made, the following kinds of 
decisions demand attention: Will one train one’s exist- 
ing staff or acquire a new staff? Will one contact di- 
rectly the author? Will one contract with a program- 
ming source, a company interested in research and de- 
velopment? Will one buy an existing company? 
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Oiie then begins to identify the marketing objectives 
j.ireparatory to setting up a distribution apparatus. One 
must decide whether one is going to be in home-study, 
in school selling, industrial training, the government 
market, in international education and in each of these 
areas, tactical decisions must be made. In home- 
study, for example, one must consider mail-order, 
door-to-door distribution, bookstores, tutoring schools, 
franchised distributors. In the school field one must 
decide whether to use direct mail, educate and equip 
an existing sales organization, or create a new one. In 
the industrial field, should one create a new company, 
work with an overseas aflfihate, or establish a joint- 
venture operation? Will the United Nations agencies 
and the various branches of the U.S. Government be 
the best approach to the international market? 

Having decided on one’s market, one then begins to 
select program content. What courses are to be pro- 
grammed? Will one specialize in mathematics, science, 
language arts, social studies, or perhaps concentrate 
on vocational training? 

It is also important to analyze curriculum emphasis: 
modem material, traditional material, or a combina- 
tion of both? There are, of course, further details such 
as determining optimum course or program size. There 
are a lot of 100-frame programs coming on the market 
now: others of 200, 300, 400, 500 and on up to 8,000 
frames. 

How will the program be presented? In. a machine, 
in a book, or some other method? If a machine, will 
it be a machine which uses paper, films, or a tape? 
If the machine is to use paper, will the program be 
on 8 V 2 " X 11" sheets, paper rolls, or paper fanfolds? If 
the machine is to use film, will it be 8mm, 16mm or 
35mm film? 

These questions may all sound very prosaic and 
mechanical but each one of the foregoing decisions 
generates other decisions. If a machine using paper 
is chosen, what kind of paper stock should be employed; 
printed on both sides; perforated paper, white paper, 
colored paper with matching complementary inks, 
perhaps, in te rms of Angstrom units? If one selects 
a machine, what will it be required to do? Record 
responses, analyze responses, branch, count errors, 
time responses? Will the program be changed on the 
basis of student responses? 

If the programmed text format is chosen, will it be 
linear or branching? If linear, should it be with or with- 
out a mask? If a mask, should it be made of paper or 
plastic? Will one choose horizontal page-turning, or a 
vertical arrangment; will the book be turned over to 
read in the reverse direction? And should the text be 
a paperback or hard cover with wire or plastic spiral? 

There are, of course, options for quasi-teaching ma- 



chines, which may consist of paper envelopes or 
boxes, response-reinforcing devices, dye-spot tech- 
niques, coated, masking techniques, mechanical punch- 
ing ... ad infinitum. 

Somewhere along the line one must decide whether 
programs are going to be consumable or non-consum- 
able, and decide whether response devices or pre- 
printed answer pads are to be made available. 

Having made all these preliminary decisions, one 
must begin to procure and sell something, programs 
and/or machines. Shall they be sold off-the-shelf, 
will one seek contracts, train programme.’’S? Then 
comes the marketing strategy. Are the programs to be 
sold as basal material; should they be sold with ancil- 
lary equipment such as laboratory devices? Will 
teachers’ manuals, language tapes, and extra tests be 
provided? 

A decision of increasingly greater importance is the 
setting up of the pre-publication testing program. 
Should the testing program be set up within one’s own 
organization or outside? Will the testing be internally 
financed or in a cooperative venture with a research 
organization, or with a school? Only after programs 
are thoroughly tested and revised are they finally 
ready for publication. 

Once programs are published, one’s work is not com- 
pleted. If your marketing results are satisfactory, one 
can then start worrying all over again: about the next 
generation machines, the next generation of program- 
ming technology, and future instructional technology in 
general. 

What about the publisher’s role in the future? To 
begin with, textbook publishing will never be the same 
again. Forgetting for the moment that there are four, 
five, or any other given number of principles of pro- 
grammed learning, may I suggest that there is one sem- 
inal factor which every publisher must now take into 
account: the learner-oriented pre-testing prior to pro- 
gram publication which is basic to the technique, 
and which will re-shape much of the textbook author’s 
thinking about his product. 

Since programmed instruction has claimed much, has 
promised much, much is quite properly being demand- 
ed of it. If programs can be designed to produce a 
specific learning objective and can climcally demon- 
strate its achievement, cannot all "jther instructional 
materials — ^including the bulk of standard textbooks — 
also be so developed? 

As an indication of what the future holds, I would 
submit the following quotation for your consideration: 

Scientifically constructed materials for learning and for 
appraisal and diagnosis, in a form so as to be ready-at-hand 
and constantly-used conveniences for both teacher and pupil, 
should be imiversally available. There must be an ‘industrial 
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revolution* in education, in which educational science anu the 
ingenuity of educational technology combine to modernize the 
grossly inefficient and clumsy procedures of conventional edu- 
cation. Work in the schools of the future will be marvelously 
though s'mply organized, so as to adjust almost automatically 
to individual differences and the characteristics of the learning 
process. There will be many labor-saving schemes and devices, 
and even machines, not at all for the mechanizing of education 
but for the freeing of teacher and pupil from educational 
drudgery and incompetence. Teachers and pupils wiU cooperate 
in fascinating efforts to develop further conveniences. The 
future school will, in consequence, be as much more efficient 
than the schools of the past as modem industry is more 
productive than the handicraft of 200 years ago. Will all this 
make education an industrialized monster, neglectful of hiunan 
values? That it can never do; the purposes of education would 
prevent it — and a major service of such devices would be to 
guide educational effort vwth ever-increasing clearness and 
adequacy toward the accomplishment of those purposes. 

As the tme complexity of educational problems is more 
understood, the need for ways more easily to handle these 
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yet larger problems becomes greater. So, increasingly it will 
be necessary (and because more necessary, more likely) that 
special devices and mechanical aids and an industrial revolution 
in education will come about. 

The foregoing quotation, which to my ear sounds 
nearly as fresh as though it had been written in 1963, 
was in fact published in a book by Pressey and Robin- 
son, “Psychology and the New Education,” in 1933. 

So, to those who might say v/e’re going too fast, per- 
haps out-running the state of the art, I would reply 
that we are too slow. If we had had a faster start 
thirty years ago, many of our recent and current prob- 
lems and some of our future problems would be behind 
us. And who knows how much closer we mi^t now 
be to world peace, prosperity, understanding? But it’s 
idle to speculate on what lost time may have cost us; 
let us simply not lose any more time. 
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XII A Point of Transition 

Wesley C. Meierheniy 

Wesley Meierheniy summarizes the present trends in programmed instruction as 
reflected in the preceding chapters. With this point of transition he offers some 
penetrating suggestions for the further developments and future trends in the state of 
the art. 



This chapter, “A Point of Transition,” is written 
from the point of view of programmed instruction as 
used in the traditional sense. There are some develop- 
ing conceptions of the word “programmed” as being 
practically synonymous with the systems approach 
which is discussed later on in this chapter. There are 
a considerable number of educational specialists who 
are referring to programmed instruction as being that 
typ^. of instruction where the entire sequence may be 
replicated exactly. In such a definition, the use of tele- 
vision, motion pictures, and similar presentations could 
qualify as being “programmed instruction.” To prop- 
erly interpret this chapter, the reader should assume 
that a program is defined as “a sequence of carefully 
constructed items leading the student to mastery of a 
subject with a minimal error. The distinguishing char- 
acteristics of programming materials is the testing pro- 
cedure to which they are subjected. Empirical evidence 
of the effectiveness of each teaching sequence is obtain- 
able from the performance records of students” (Ely, 
1963). The point of transition, therefore, is the pres- 
ent state of the art as reflected in the preceding chapters 
along with suggestions as to possible future trends and 
developments. 

Programmed instruction brings into clearer focus 
than any other preceding change in education the 
basic question of the purposes of education in a 
democracy and the way in which these purposes are 
developed. The reason for this philosophical con- 
frontation is that the very first element in programmed 
instruction deals with the clear identification of objec- 
tives in some type of behavioral form. Those who deal 
with programmed instruction, therefore, face the neces- 
sity of determining from some sources what should be 
accomplished in the total educational program as well 
as how individual bits and pieces of the educational 
program might be pre-determined and fitted together. 

Traditionally, there is a feeling that purposes for a 
school system grow out of the wishes and desires of 
the citizens of the local community. Historically, some 
attempts have been made to bring this question into 
clear focus. One such is the book by George Counts 
(1932) entitled “Dare the Schools Build a New Social 
Order?” The strong opposition with which this thesis 
was greeted, even in the depths of the depression, is 



an indication of how antagonistic many people are, 
including professional educators, to the idea of the 
school systems playing a carefully calculated role in 
the renewal or re-direction of our society. 

Lest one feel that this resistance to the planned role 
of schools was true only of the depression days, some 
recent conferences of educators have indicated a 
laissez-faire role for the schools. For example, when 
new directions in education are proposed by someone, 
there is an immediate question from many as to 
“whose directions?”. The implication is that the schools 
must wait until there is sufficient crystallization within 
a given community in order that new directions can be 
taken. 

The basic dilemma may well explain the rapid 
acceptance of programmed instruction in business, in- 
dustry, and the military as contrasted with the lack of 
acceptance in public education to which several writers 
have referred in earlier chapters. In a business enter- 
prise, the instructional objectives are determined by the 
firm involved which is responsible to no one except its 
own Board of Directors. The ease of defining these 
objectives makes it possible, therefore, to develop train- 
ing routines which do not raise any basic philosophical 
questions. Much the same thing is also true in the 
case of the military where there is a reasonably clear 
function to be performed by training which does not 
get involved in the whole matter of public policy. 

The development of programs for use in public 
schools reflects the dilemma which is faced by edu- 
cational institutions. Courses in mathematics, foreign 
languages, and other subjects of a similar nature, where 
the objectives are primarily intellectual and skill acqui- 
sition, have received much attention by programmers; 
on the other hand, the social sciences and, to a degree, 
the language arts have not had similar efforts. Perhaps 
this basic difficulty of a philosophy is related to some 
of the subject matter areas where the lack of agree- 
ment as to what constitutes appropriate behavior as a 
citizen in regard to his local community, the state, the 
nation, and the world comes into clear focus. 

A first and overriding consideration in the field of 
programmed instruction is, therefore, the basic problem 
of stating objectives for education beyond general plati- 
tudinous statements and to more specifically and clearly 



state objectives within the context of a dynamic demo- 
cratic social order. In the preceding chapters, there are 
a number of indirect references to this problem but not 
as clear a recognition of it as must be made if pro- 
grammed instruction is to make much greater con- 
tributions. 



Programmed instruction is forcing a systems ap- 
proach to instructional operations. There are varying 
ways of perceiving and defining a system. A part of 
the difficulty of definition grows out of the question 
as to what a system is and what is a sub-system as well 
as the question as to whether there is such a thing as 
a closed system. Almost any occurrences, such as 
political, economic, religious, or other happenings, have 
implications for “an educational system,” Nevertheless, 
in spite of the difficulty of definition, it is possible to 
direct the discussion of a school system to individual 
schools or attendance units within the system to edu- 
cational programs within each of die systems and on 
to other more minute sub-systems within the school 
system. The reason that the whole systems approach 
is receiving so much attention is that programmed 
instruction should be viewed and conceived as a part 
of the system and introduced and utilized in such a 
way that it does not dislocate other parts of the 
whole. The systems approach is related to the previous 
discussion of purposes or philosophy since the devel- 
opment of the sub-systems to be presented by pro- 
grammed instruction must be conceived as an integral 
part of a larger system. 

A difficulty presented by the systems approach is 
one of keeping the whole enterprise flexible to meet 
new demands rather than permitting it to become fixed 
and static. To a considerable degree, educational enter- 
prises in the past have become too rigid and may do 
so even with programmed instruction unless everyone 
connected with the educational enterprise recognizes 
the need for constant inventiveness, adaptation, eval- 
uation and revision of system components. 

The application of the systems approach to social 
institutions presents greater problems than its applica- 
tion to an engineering problem where the systems idea 
was first developed. For example, the perception of a 
system for orbiting a satellite requires provisions for 
certain contingencies. There must be mechanisms for 
making adjustments during each stage of the flight of 
the rocket brought about by a m.agnitude of forces 
which can be identified but the extent of them cannot 
be predicted in advance and, therefore, certain flexi- 
bilities must be built into the system. There are, how- 
ever, two Doints which can be identified which are fixed. 
They are the launching pad and the eventual orbiting. 
Contrasted with these fixed points, however, is a social 
institution which is an integral part of a cultural envi- 
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ronment which is always changing in ways that are 
difficult to measure. Therefore, social institutions 
operate in a fluid situation — the various aspects of 
which are difficult to define at any given moment and 
where it is difficult to determine the dynamics among 
the various aspects of the culture. This is not to say 
that it is impossible for social institutions, such a.s edu- 
cational enterprises, to identify and to plot the various 
forces which play upon them, but it does indicate the 
necessity for increased research in the social and beha- 
vioral sciences. Here again the implications for pro- 
grammed instruction are clear because of the demand 
that such instruction be considered as an integrated 
part of a complex operational system. The proponents 
of programmed instruction have attacked the problem 
of stating objectives in behavioral form and certainly 
have made more progress than ha.s ever been made 
before in this area. 

There is evidence that task analysis has received 
considerable attention in programmed instruction. The 
attention grows out of the necessity for studying how 
a task is performed or should be performed. In the 
case of new instructional demands, the task analysis 
calls for considerable specificity on the part of those 
who are proposing the new educational requirements. 
For example, the task analysis specialists have every 
right to require that those responsible for proposing 
the new instructional sequence verify the task to be 
performed because the whole programming sequence 
proceeds from the task analysis and unless the task 
analysis was done properly the outcome of the training 
program may be far off the target. 

As an integral part of task analysis, there must be 
a conversion into some type of behavioral objectives. 
A great deal of activity related to programmed instruc- 
tion has been in this direction. Programmed instruction 
has shown the necessity for a departure from loose 
general objectives to those which can be stated opera- 
tionally. We are still stifled at the moment, however, 
by our inability to place educational objectives into 
some kind of taxonomy and then to transfer the result- 
ing objectives into acceptable behavioral terms. An 
integral part of the programmed instmetion movement 
has been to give attention to the specific outcomes of 
instruction, but still more needs to be done. 

A further point brought on by programmed instruc- 
tion is that pertaining to testing and evaluation. In 
the preceding discussions it has been clear that we are 
attempting to determine what instruction should accom- 
plish. We are beginning to mere nearly define what 
we hope will be accomplished by instructional pro- 
cedures, but are now faced with the necessity of devel- 
oping valid measuring instruments to determine if the 
objectives have been reached. Although the testing 
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movement has made great strides in the United States 
in the last one-half century, anyone who has conducted 
research studies has been struck by the lack of scien- 
tific instruments to measure changes which have re- 
sulted from specified variables. Programmed instruc- 
tion Dresents additional problems of evaluation, such 
as, should criterion frames also be used as test frames? 
Should we expect that as learners pursue a program 
they will be able to perform related tasks other than 
the precise ones which the program has prepared them 
to perform? There is also the related question as to 
whether tests used in programmed instruction have 
quite different bases and purposes than those used 
previously. For example, in the field of achievement 
testing, there is the principle of “preserving the inde- 
pendence of the measure”, which means that the indi- 
vidual developing the test questions is aware of what 
will be taught but does not know specifically what 
instructional procedures are being, or have been taken. 
In the case of programmed instruction, there is almost 
always an insistence that the person developing the 
test items know exactly what is planned in the pro- 
grammed sequence. The fact is that these two tests 
often are prepared by one and the same individual. 
One has, therefore, a fundamental difference in per- 
spective between the traditional testing approach and 
what takes place in programmed instruction. 

A related question is whether the test or criterion 
items should be specific in nature, or whether they 
should be drawn from a family of items, any one of 
which would be an index as to whether or not the 
learner would be able to perform the designated objec- 
tives. In other words, there is the question as to 
whether the test items should be as nearly like the 
frames in the program or whether there was prior 
activity on the part of both the programmer and the 
test maker in the specification of objectives related to 
a family of tasks and thus of test items. 

There is still a considerable difference of opinion as 
to what can be programmed now and what we may 
eventually be able to program. Opinions range from 
an extreme that anything can be programmed where 
the objectives can be stated in behavioral tenns to the 
opposite end of the spectrum to those who believe that, 
for the present, programming is limited to rather highly 
specific and narrowly defined tasks. As alluded to 
earlier, it may well be that part of the difficulty in 
programming all subject matters is related to a philo- 
sophical dilemma rather than a programming diffi- 
culty. For example, if it should be possible to define 
what is appropriate behavior for persons of different 
races, colors, and creeds to display toward each other, 
one well might be able to program the appropriate 
sequences leading to the desired behaviors. The diflB- 



culty at the moment is to obtain any kind of universal 
agreement as to what represents acceptable behavior 
in various social situations. In the transition period 
we shall no doubt give much attention to varieties of 
outcomes which can be brought about by programmed 
instruction including such possibilities as values, reason- 
ing, and creative thinking. 

Much of the activity of experienced programmers 
has been related to the development of programming 
from certain theoretical perspectives. A basic dichot- 
omy is said to exist between linear and branching 
programs. If there were marked differences in the 
early history of programming they seem to be dis- 
appearing somewhat. What does persist, however, is 
a philosophical difference as to the most economic and 
propitious means of reaching instructional objectives. 
This debate is really one related to theoretical consid- 
erations, the one pretty generally laboratory based and 
the other intuitively and empirically based. A resolu- 
tion of the question may once again be in terms of 
what are the ultimate objectives, after which the pro- 
cedural matters may be solved. The conflict perhaps is 
also related to the question as to whether the approach 
in programming should be from the “micro” or the 
“macro” point of view. The one approach would re- 
quire minute analysis of each stage in the learning 
sequence whereas the other would start with broad 
overall objectives and attempt to identify the minimum 
essentials to be learned. The former might assume the 
necessity of a large number of small steps, while the 
latter approach might well concern itself with broad 
key learnings which could, when synthesized, make 
possible the performance of the specified learning task. 

As programming moves forward there will no doubt 
be added explorations of various grid and matrix 
approaches in order to give clues to both the writer 
and evaluator as to the key points in content and possi- 
ble inter-relations among them. Some programmers 
are, in fact, dealing with three-dimensional matrix 
models as a means of achieving greater scientific analy- 
sis of the content. During the transition phase, there- 
fore, we probably will see much more attention being 
given to the analysis of the subject matter structure by 
means of various types of models. 

One of the most significant contributions made in 
the programmed instruction movement is that of the 
inter-relations which must exist among a programming 
team including content, methodology, and media spe- 
cialists. It is obvious that someone on the team must 
be a specialist in the content. The insight of a peculiar 
kind of scholar is required who is capable of examin- 
ing his discipline in such a way so as to identify the 
basic aspects of the structure upon which the mastery 
of the discipline depends. There is still some difference 



of opinion, however, as to whether structure does exist 
at all and, if it does, whether each branch of knowledge 
has its own unique structure. There are sufficient clues 
available to lead one to more extensive explorations in 
this area. For example, in the field of mathematics it 
now appears that there are three fundamental princi- 
ples involved, whether one is considering the most 
elementary number work in the first grade or the high- 
est level graduate course, which are (1) set of num- 
bers, (2) operations on these numbers, and (3) prop- 
erties of these operations. As simple as these three 
principles seem to be, it has been only recently that 
scholars in the field of mathematics have communi- 
cated this simplification of the discipline. It is neces- 
sary, therefore, to have involved on the programming 
team a representative of the subject matter area which 
is to be programmed. 

A second equally as significant fact as the discovery 
of the essential elements of the content are the strate- 
gies and methods of teaching them. In the history of 
programming, more contributions have probably been 
made at this point than at any other place. Because a 
great deal of the programming skill has been developed 
in psychological laboratories, much of the methodology 
has come from learning theorists. As we move forward, 
we also are likely to involve the trained and insightful 
teacher who has intuitively worked out some methods, 
which have proved to be successful for her. For exam- 
ple, many of the national curricular movements, such 
as the BSSC biology course, discovered the genius of 
classroom teachers for advising content specialists on 
the approaches which will produce the most rapid 
learning on the part of students. Because most psy- 
chologists have worked in laboratory situations, their 
understandings of the kinds of variables which play 
upon ordinary classroom situations often are not ex- 
tensive. It is for these reasons, tlierefore, that the 
methodology with which the content is presented in the 
program requires the joint efforts of both theorists and 
practitioners. 

A second problem of method is related to feedback 
from the learners. In some preliminary work which 
has been done there are indications that on material 
in which the learner has had some prior familiarity, 
the learner is more able to identify the necessary strate- 
gies by which he should learn than is the subject mat- 
ter specialist. Taken at face value this would seem to 
be a denial of the premise that the scholar in the dis- 
cipline is best suited to identify essential elements of 
content. It must be kept in mind, however, that the 
studies which have been completed involved subjects 
who had considerable information about the area even 
though it was not well organized and in some cases 
not even known to exist on the part of the subjects. 
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The point is, therefore, that we may well be able to 
observe learners in order to give us assistance in both 
the content and presentation decisions, but where the 
material covers new material we will have to continue 
to obtain much of our insight from the content spe- 
cialists. 

A third member of the programming team whose 
role is just now evolving is a person who is a specialist 
in the identification of the experiences which are needed 
by learners in order to satisfy the objectives of the 
programmed sequences. The early history of programs 
was that they were almost entirely of a verbal nature 
both so far as the stimulus and response frames were 
concerned. Perhaps this was to be expected since the 
specialists in the field of human learning have been 
concerned primarily with verbal materials although 
some prominent human learning theorists have at- 
tempted to extrapolate findings from animal labora- 
tories and apply them to human learning. The research 
conducted by learning theorists has emphasized verbal 
learning and included ever 3 rthing from nonsense sylla- 
bles to meaningful verbal information. Although the 
implications of nonsense, meaningful, and serial learn- 
ings have implications for regular classrooms, these 
variables are somewhat limited in terms of the complex 
learning situations found in classrooms. It is quite 
evident, therefore, why programs coming from psy- 
chology laboratories have been primarily of a verbal 
type. As programmed instruction moves more and 
more into the main stream or regular classroom activi- 
ties, it is quite likely that a much broader range of 
stimuli will be used in order to produce certain leam- 
ning outcomes other than verbal ones alone. In this 
period of transition, therefore, we wiU be faced with 
extensive work in the area of media, both for stimulus 
and response purposes. An individual whose role is 
that of message designer for all types of media including 
visual, auditory, and verbal approaches will be needed 
as a member of the team. 

During this period of exploration and experimenta- 
tion, we are likely to see continued attention given to 
a wide variety of factors within the programs them- 
selves. One such variable deals with the length of the 
program. Even though it might be possible to program 
a semester or a complete course in a subject matter 
course area, we may well find some obstacles to the 
adoption of such courses and other approaches will 
have to be attempted. For example, although it may 
be advisable or practical to program units of a course 
or a complete course in a subject matter course area, 
we may well find some obstacles to the adoption of 
such courses and other approaches will have to be 
attempted. We may have to program units of a course 
or a certain discrete bit of subject matter which is 
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really not in a sequential pattern such as the appropri- 
ate role of the slide rule in mathematics or science. 
A part of this problem might be related to the diffi- 
culties inherent in the introduction of any new practice 
rather than a problem innate in the instructional 
sequence. 

Another variable in programs pertains to the modes 
of presentation. Considerable success has been achieved 
for some programmers in the presentation of stimulus 
material by such a medium as television. Although this 
practice seems to be quite the opposite of one of the 
avowed advantages of programmed instruction, there 
does appear to be some maximum time which most 
learners require in order to assimilate the stimuli and 
make appropriate responses. In spite of the known 
variations of all kinds among learners, they may not 
be quite so diverse but that a well developed mass 
presentation may be possible. In addition to television, 
such approaches as microfilm and motion pictures, 
including 8 mm, represent alternate possibilities to the 
traditional text or printed approach to programming. 
There are other systems being developed which are 
complex in their operation, frequently including com- 
puters, with which much more experimentation can be 
expected. 

Although there has been a good deal of research in 
the matter of overt vs. covert responses, the period of 
transition will probably see a better reduction of this 
problem. Here, as in some of the previous problems 
discussed, it might be found that the type of responses 
are related to such matters as the prior experience of 
the learners and the purposes to be achieved through 
the instruction. 

There also will be studies dealing with such matters 
as the performance patterns and rhythms of the learners 
themselves. For example, there are some studies on 
the physiological aspects of human beings which indi- 
cate that there are certain periods of higher productivity 
as well as lower productivity depending upon the time 
of the day. This factor may also be related to such 
matters as fatigue and the length of rest periods taken 
by subjects pursuing programmed instruction as con- 
trasted with regular school schedules. The above 
physiological and emotional factors will no doubt be 
researched much more in the future. 

Related to the response phase of programmed 
instruction is the basic question of reinforcement. 
Although reinforcement was at one time considered 
essential, research findings have not always borne out 
the need for immediate or constant reinforcement. 
Because the principle of reinforcement has had such 
a central role in programmed instruction, much more 
study is needed of this factor. 



The use of progranuned instruction in public school 
situations has been quite disappointing. The reasons 
may lie in a number of directions, some of which may 
be overcome during this transition period. The first 
is the increased consideration of the problem of inno- 
vation along with diffusion of new products among a 
large number of users. It is evident now that innova- 
tion is a separate phase from diffusion. Programmed 
instruction is an innovation and various improvements 
in programmed instruction will also be innovations of 
a type. Since the innovation of programmed instruction 
has taken place, we need to focus more on the area of 
diffusion. Without going into all of the ramifications 
concerning diffusion practices, the following steps gen- 
erally seem to be involved: 

1. Stimulate interest in programmed instruction 
through publications, meetings, discussions, and other 
similar approaches. Along with motivational activities 
there must be a cultivation of the environment so that 
changes and modifications a:C expected by those con- 
cerned with the system. 

2. Arrange a visit to a situation in which pro- 
grammed instruction is being used. If this is in the 
field of education, it should involve teachers talking to 
other teachers and observing programmed instruction 
being used with children, so that research results might 
be checked against actual results as reported by col- 
leagues and the learners. 

3. Organize a pilot project. The project should 
probably be of limited scope initially, so that it can be 
carried out without involving a large number of other 
people within the system. This may mean individual 
units rather than whole courses. 

4. Give a great deal of help during the tryout 
period, including the generous use of consultants and 
specialists. In spite of the fact that many of the changes 
involving programmed instruction do not seem compli- 
cated, the people using programmed instruction for the 
first time need a great deal of assistance. 

5. Give considerable support and recognition to 
those individuals participating in the pilot project. 
This includes individual praise as well as public recog- 
nition. 

A second problem in the widespread acceptance of 
programmed instruction may be the difficulty in know- 
ing how to select good programs. Although there have 
been some criteria developed in regard to this matter, 
much more still needs to be done. Research results 
alone, even though statistically significant, are not 
sufficient to bring acceptance on the part of many 
people in the field of education. We need to translate 
these research results into behavioral pattern, so that 
teachers may recognize the terminology and be able to 
apply the findings to their own situations. 
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Publishers of programs have not been too helpful 
in regard to criteria and standards. In many instances 
they have little, if any, information about the levels at 
which this program might be used and the effectiveness 
of the programs when utilized at the specified levels. 
In addition, publishers have not supplied related mate- 
rials which would help administrators and teachers to 
know how the materials should be used in the class- 
room. A part of the success of the National Science 
Foundation in initiating new national curricula is due to 
putting materials in the hands of teachers and also 
helping them to learn how to use them. The prepara- 
tion of teacher handbooks with accompanying illustra- 
tive materials will no doubt be attempted in the future 
as means of showing practitioners how to use pro- 
grammed instructions. 

All of the current work in programmed instruction 
as well as most educational research in the United 
States is represented by a behavioristic approach. There 
is no doubt but that a good deal of progress has been 
made in the field of instructional technology as a result 
of such an attack. An appropriate question is whether 
our current attempts to achieve a break-through leading 
to an entirely new conception of the learning process 
are being inhibited by our current models, some of 
which have been in existence for decades. There is a 
body of information just now coming from laboratories 
around the country leading to considerable specula- 
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tion that the learning act may be much more closely 
related to chemical changes in the brain and nervous 
system than had been hypothesized heretofore. The 
work at the University of Rochester with the planarian 
worm and the work at the University of California at 
Los Angeles with electrodes appropriately placed in 
the brains of primates strongly suggest the thesis con- 
cerning chemical changes during the learning act. 

There needs to be attention given to the possibility 
that physiological factors may be much more involved 
in learning than generally thought to be the case. 
Chemical changes may give us the explanation as to 
what actually happens in the learning process and thus 
not make it necessary to make second hand observa- 
tions with generally crude tools and instruments. 

There can be no question but that programmed in- 
struction has had a greater impact on the field of 
teaching and learning than has any one innovation in 
educational history. It is clear from the preceding 
chapters, as well as this chapter, that we are still a 
long way from having all the answers that are needed 
to make final decisions. It appears that we are at the 
point of an e^losive take-off which will involve mas- 
sive amounts of basic research, developmental activi- 
ties, and field studies. These will be combined in such 
a way that the true instructional technology so well 
postulated in the first chapter of this volume will be 
well on the way to being accomplished. 
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specializing in military training programs. While in the 
Air Force, he assisted in the development of atomic 
weapon capabilities and had responsibility for the training 
of both ground and air crews. 

F. ELAINE DONELSON, M.S., Experimental Psychol- 
ogy. Currently a candidate for her Ph.D. in Social Psy- 
chology, at Cornell U, she was previously associated with 
tlie Martin Company, Orlando, Fla., as a member of the 
Human Factors Section. She is co-author of Programmed 
and Conventional Instruction — An Experimental Evalua- 
tion (1963). 

SISTER MARY DOROTHY, M.A., George Peabody 
College, Nashville, Tennessee, is Principal of St. Martin 
Hall, Campus Laboratory School, and a Professor of 
Geography, Our Lady of the Lake College, San Antonio, 
Texas. She is a Sister of Divine Providence and has had 
experience as a teacher and an administrator in several 
schools in Oklahoma and in Texas. 

CARLTON B. DOWNING, Ed.D., is Professor of Edu- 
cation and Research, Trinity University, San Antonio, 
Texas. Formerly a public school administrator and mem- 
ber of Gilmer-Aiken Commission that reorganized Texas 
System of Public Schools, he is currently Vice President, 
National Society for Programmed Instruction and a 
teacher, lecturer and contributor of articles and reviews 
on programmed instruction. 

WALTER E. DRISKILL, M.A., Southwest Texas State 
College, is a Training Specialist in the Training Evaluation 
Branch, Operations, Lackland Military Training Center. 
In addition to training evaluation and research work, he 
is project officer of the LMTC programmed instruction 
effort. He is currently completing requirements for the 
Ph.D. at the University of Texas where his major interest 
has been learning theory and how it relates to the develop- 
ment and use of programmed materials. 
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LEWIS D. EIGEN, New York City, is the Executive 
Vice President of the Center for Programed Instruction. 
Along with Stuart Margulies, he edited the book. Applied 
Programmed Instruction. He is the author of Several pro- 
grams in high school mathematics and elementary school 
science, and has published many research studies in the 
field of programmed instruction. He has lectured on pro- 
grammed instruction at Teachers College, Columbia Uni- 
versity; the University of California, Berkeley; the Uni- 
versity of Chicago; and many others. 

HENRY C. ELLIS, Ph.D., is associate professor of Psy- 
chology at the University of New Mexico in Albuquerque. 
He has previously taught at Washington University, Sfc 
Louis, and is the author of several publications on human 
learning and transfer. He is presently consultant on pro- 
grammed learning to several agencies and is an Asso- 
ciate Editor of Perceptual and Motor Skills. 

JAMES L. EVANS, Ph.D., University of Pittsburgh, is 
Vice President of Teaching Machines Inc., in Albuquer- 
que, New Mexico. He was Research Assistant at the 
University of New Mexico, American Institute for 
Research, and at the University of Pittsburgh. Since 
1958 he has contributed numerous papers to the field of 
programmed learning, including his doctoral study. An 
Investigation of “Teaching Machine” Variables Using 
Learning Programs in Symbolic Logic, and a review chap- 
ter on Programming in Mathematics and Logic for the 
second volume of the NEA-DAVI publication entitled 
Teaching Machines and Programmed Learning. 

JOHN R. EVERETT, Ph.D., Columbia University, is 
President of Encyclopedia Britannica Press, Chicago and 
is Senior Vice President of Encyclopedia Britannica. He 
was the Chancellor of The City University of New York, 
1960-62 and the President of Hollins College, 1950 60. 
He is the author of Religion in Human Experience (1950) 
and of articles and reviews on. religion in economics, 
religion, philosophy, economics, etc, 

J. E. GERBER, JR., M.A., is a civilian Education Spe- 
cialist at the United States Army’s Medical Field Service 
School, Fort Sam Houston, San Antonio, Texas. He is also 
engaged in additional studies at The University of Texas. 
He was formerly associated with a radiobiology research 
group (1956) at the United States Air Force School of 
Aviation Medicine. 

JAMES E. GILBERT, M.A. Boston, Mass., is Asso- 
ciate Professor and Director of the Division of Pro- 
grammed Instruction Technology, Northeastern Univer- 
sity. He was previously Staff Educational Psychologist and 
Research Psychologist for the Department of Defense, 
Washington, D. C., as well as consultant to the Schcc! 
Mathematics Study Group Programmed Learning Project, 
Stanford University. 
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merly resident consultant on training for Beii Telephone 
Laboratories. He is the author of numerous articles on 
educational technology and editor of Mdthetics JourneL 

DANNIE M. GILLHAM, M.A., is an Instructional Pro- 
grammer at Amarillo Technical Training Center of Air 
Training Command writing in the Administrative field at 
Amarillo AFB, Texas. He previously taught at West Texas 
State University and in the Amarillo public schools. 

LEO S. GOLDSTEIN, Ph.D., is Senior Research Asso- 
ciate in the Institute for Developmental Studies & Assistant 
Research Professor, Department of Psychiatry, New York 
Iviedicai College, New York City. He v.'ss formerly the 
Research Director of the Center for Programed Instruc- 
tion. He is co-author of a programmed textbook on de- 
scriptive statistics. 

J. L. GREEN, Lt. Colonel, USAF, is Chief of the Tech- 
nical Writing and On-the-Job Training Division in Head- 
quarters, Air Training Command. He has taught in both 
technical and flying training courses, and has served as an 
industrial consultant in the area of training management, 

GEORGE L. GROPPER, Ph.D., is Associate Director 
for the Training and Education Program at the A.nierican 
Institute for Research, Pittsburgh, Pa. He is the author 
of a number of research publications on the application 
of programming techniques to verbal and visual materials 
presented over television. 



ROBERT F. HARRIS, B.S.; is presently Chief, Instruc- 
tion, Measurement and Training Materials, 3345th Tech- 
nical School, Chanute AFB, 111. He has had extensive 
background teaching experience in secondary schools in 
Illinois. He has been an ciiucational specialist with the 
Air Force for 23 years. As staff project officer; ha super- 
vised ihe nmny studies of prcgramnied instruction projects 
at Chanute. 



JOHN D. HATHAW A.Y , M.B.A., is a member of the 
facaity of the Business Administration Department, San 
Angelo College, San Angelo, Texas. He previously taught 
at the University of Illinois. He is the author of the United 
States Air Force On-the-Job Training Manual, Legal Spe- 
cialist. 

ALBERT E. HICKEY, Ph.D., is President of ENTE- 
LEK Incorporated, Newburyport, Mass. He was formerly 
with the Office of Naval Research, and the General Dy- 
namics and Itek Corporations. He has been a lecturer at 
American and Northeastern Universities, and was director 
of the Automated Teaching Project at Northeastern. He is 
co-author of Programmed Instruction in Probability and 
Statistics, (2961), PERT AND CPM (1962), Inventory 
Management (1965), and oiuci programmed instruction 
courses and of research papers in learning and programmed 
instruction. 



THOMAS F. GILBERT, Ph.D., creator of “Mathetics” 
is presently a consultant to U.S. Public Health Services. 
Also was National Science Foundation Post-Doctoral Fel- 
low at Harvard University from 1958-1959. He was for- 



MIRIAM W. HIESTER, M.A., is presently Educational 
Specialist in the Medical Field -Service School, Fort Sam 
Houston, San Antonio, Texas. She has written article.s 
for the Folklore Society and manuals in the field of mili- 
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tary medicine. She has been teaching College English for 
the San Antonio College for fourteen years, presently for 
the Evening Division. 

HERBERT H. HUGHES, Ph. D., Florida State Univer- 
sity, is Assistant Professor of Psychology and Co-Director, 
Bure.'-u of Research. Colorado State College, Greeley, Colo- 
rado. He was an NIMH fellow at Florida State University. 
He has taught courses, conducted research, and served as 
a consultant in programmed instruction, measurement, and 
research design. 

ROGER A. KAUFMAN, Ph.D., is Senior Scientist, Bolt, 
Beranek & Newman, New York City. Previously he was 
Manager, Training Systems Analysis for U.S. Industries 
Educational Science Division and Head, Personnel and 
Training Group at rriartin Company, Baltimore. He is 
also a consultant to USAF Air Training Command and the 
Venezuelan Government. 



CHARLES M. KEYS is a Programmer and Publications 
Technical Specialist at General Dynamics/ Convair, San 
Diego, California. He as.sisted in the development of the 
programming capability for Convair in 1961 and has 
since written and edited numerous programs for industrial 
and ieehniCttI training in addition to his work on the 
Value Control series. 



DAVID J. KLAUS, Ph.D., Pittsburgh, Pennsylvania, is 
Director of the Training & Education Program at the 
American Institute for Research. He has been active in 
research on programmed instruction since 1958. He pre- 
viously taught courses in experimental and educational 
psychology at the University of Pittsburgh. He is the 
author of numerous research reports and articles dealing 
with programming technology and new applications for 
programmed aiaterials. 
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Indiana, and UCLA Executive Program graduate, is prcsl- 
deiit oI Training Systems, Inc. He previously was director 
of the Logistics Laboratory at Litton Industries where he 
was a member of the executive staff. Among the manly 
programs Mr. Knight has authored is a program in Man- 
agement Fundamentals and Principles of Planning. In 
November 1961, Mr. Knight presented the paper Auto- 
mated Instruction in Industry at the 14th Annual Gover- 
nors Conference in Los Angeles, California. 



ROBERT LA FONTAINE, San Antonio, Texas, is the 
chairman of the Social Studies Department, Gamer Junior 
High School, and teaches ninth grade World History and 
World Geography, He received his B.A. degree from St. 
Mary’s University and has done graduate work in the 
field of education at Trinity University where he became 
interested in programmed instruction. 



JACKIE LANG, TSgt, USAF, is with the USAF Marks- 
manship School, Lackland AFB, Texas, He has been an 
Air Force Instructor for over 14 years and has attended 
several Military Instructor Courses and the Air University. 
He has pre'/iousiy laught Basic Military Training subjects 



and has been on the teaching stafi of the Instructor Train- 
ing School and the USAF Marksmanship School. 

MAURICE A. LARUE, JR., M.S. in Physics, Ford- 
ham University, is Chief, Human Factors Section, Martin 
Co., Orlande, He was previously associated Wlul other 
aerospace and electronic firms in the capacity of Human 
Factors specialist. He is co-author of a programmed text 
Fundamentals of Computers (1962) and Programmed and 
Ccnvcniioncl Instruction — an Experimental Evaluation 
(1963), and is author of many articles on Human Factors 
and Human Engineering. 

CHARLES G. LATTERNER, M.A. is manager, train- 
ing, Federal Electric Corporation (ITT), Paramus, New 
Jersey. He is responsible for developing and implementing 
technical, electronic and other programs for industrial and 
military training. He has written and supervised produc- 
tion of many programmed texts including. Transistors, 
PERT, PERT/Cost, Mathematics for Electronics and 
others. 



JOHN M. McKEE, Ph.D., is Director of the Experi- 
mental Project in Education and Rehabilitation at Draper 
Correctional Center, Elmore, Alabama. He was formerly 
Director, Division of Mental Hygiene, Alabama State De- 
partment of Health. He is a member of the American 
Psychological Association. 

JOSEPH M. MADDEN, Ph.D„ Lt. Colonel, USAF, is 
Assistant Chief and Research Psychologist, Occupational 
Structures Division, 6570th Personnel Research Labora- 
tory, J^ackland AFB, Texas. He teaches at St, Mary’s 
University, San Antonio, Texas on a part-time basis. For 
the past five years he has been engaged in research on 
human judgment as applied to job evaluation, job analysis, 
and other phases of personnel management. 

P.ORERT F. MAGER, Ph.D., is Director of Research 
and Development and Technical Services of learning Sys- 
tems Institute, Paris, France. He previously was Senior 
Scientist of the Automated Instruction Project at Varian 
Associates, and Senior Scientist at the Human Research 
Unit, Fort Bliss, Texas. He is author of Preparing Objec- 
tives for Programmed Instruction (1961) and of articles 
on learning and programmed instruction. 

STUART MARGULIES, Ph.D., Columbia University, 
is currently Director of Special Projects at Basic Systems, 
Inc. He has worked at the Naval Training Device Center, 
the Center for Programmed Instruction, and Automated 
Instructional Materials. He has authored and edited more 
than 15 instructional programs, including several audio- 
visual courses. With Lewis Eigen he is co-editor of the 
book. Applied Programmed Instruction, and is author of a 
variety of research papers and general articles in the field 
of programmed instruction and learning. 

GIFFORD M. MAST, B.S. Chicago (Phi Beta Kappa); 
M.S., Han'axd; is President of Mast Development Com- 
pany, Davenport, Iowa. While he has taught physics and 
mathematics at St. Ambrose College, his background is 



largely in directing the development of industrial products 
and specialized optical and training equipment. 

SYLVIA R. MAYER, Ph.D., L. G. Hanscom Field, 
Bedford, Massachusetts, is a Research Psychologist at 
Electronic Systems Division, Air Force Systems Command. 
She was previously in teaching and research at Massachu- 
setts Institute of Technology and Boston University. She 
is author of articles on photo-intelligence and on auto- 
mated training in electronic systems. 

JAC D. MEACHAM, La Mesa, California, is Publica- 
tions Editor at General Dynamics/ Convair. He is Presi- 
dent of the San Diego Chapter of the NSPI. He has 
written over a dozen programs; the most widely used is 
the Programmed Instruction-r-Training Manual. He has 
written many articles and reviews. He is on the faculty 
at the University of California extension schools. He is 
currently training 54 aerospace employees in the process 
of program writing. 

WESLEY C. MEIERHENRY, Ph.D., is Assistant Dean 
of the Teachers College, University of Nebraska, Lincoln, 
Nebraska. He has been a public school teacher, superin- 
tendent of schools, and a staff member in higher educa- 
tion. He has written many articles and served as a con- 
sultant to many different groups in the general area of 
newer media including programmed instruction. He is a 
member of the APA, AERA-DAVI Committee charged 
with the responsibility for developing standards for pro- 
grammed instruction and was on the staff of the National 
Education Association Meeting on the Selection and Use 
of Programmed Instruction in Detroit, Michigan, during 
the summer of 1963. 

JAMES E. MELTON, T/Sgt, USAF, Instructional Pro- 
grammer, Instructional Systems Division, Headquarters 
Lackland AFB, Texas. Since 1948 he has been an Air 
Force Military Training and Technical Instructor. He 
received his training in programming at Learning, Inc., 
Tempe, Arizona, under the direction of Dr. Jerry G. 
Short. 

JOHN A. MODRICK, Ph.D., Fort Collins, Colorado, 
is Assistant Professor of Psychology, Colorado State Uni- 
versity. He was previously Chief, Maintenance Training 
Section, Training Research Branch of the Behavioral 
Sciences Laboratory, Wright-Patterson AFB, Ohio. He 
is the author of several reports on learning and training. 

CHARLES S. MORRILL, M.A., Columbia University, 
is a member of the Technical Staff at the Mitre Corpora- 
tion. He previously has been associated with Tufts Uni- 
versity, RCA, and Bolt, Beranek and Newman, Inc. He 
is the author of Teaching Machines: A Review (1961) 
and of articles in the areas of engineering, psychology and 
education. 

JOSEPH E. MORSH, Ph.D., is the Chief, Job Anal- 
ysis Branch, 6570TH Personnel Research Laboratory, 
Lackland Air Force Base, Texas. He previously taught at 
the University of British Columbia and at the University 
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of Hawaii. He has published many articles on instructor 
effectiveness and occupational analysis. 

JEAN S. MOYE, B.A., College of St. Rose, Albany, 
New York and Cornell School of Labor and Industrial 
Relations, formerly with the Knolls Atomic Power Labo- 
ratory, General Electric Co., is now Personnel Director of 
United Services Automobile Association, a reciprocal inter- 
insurance exchange, for officers of the Armed Services, a 
contributor to AMA publications, and a faculty member of 
Incarnate Word College. Mrs. Moye is also a member of 
an Insurance Industry Personnel committee. 

EDMUND C. NEUHAUS, Ph.D., North Bellmore, 
New York, is Director of Psychological & Educational 
Services at Human Resources Foundation, Albertson, Long 
Island. He also is a psychotherapist in private practice, 
treating emotionally disturbed children and adolescents. 
He was previously Executive Director, Nassau County As- 
sociation for Help of Retarded Children and Adjunct 
Assoc. Professor, C. W. Post College of Long Island Uni- 
versity. He is well known for research study published as 
Personality Study of Asthmatic and Cardiac Children 
(1958) and many publications on mental retardation. 

GABRIEL D. OFIESH, Ed.D., Colonel, USAF, is 
Director, Training Methods, Assistant Chief of Staff 
for Training Development at Headquarters Air Training 
Command, Randolph AFB, Texas. He was formerly pro- 
fessor of Psychology and Director of Leadership Studies 
at the USAF Academy; has taught extensively in the 
field of Educational Psychology, was assistant professor of 
psychology at Butler University, IndianapoUs. He was 
founder and first president of the National Society for 
Programmed Instruction (1962-1963). 

JOHN R. OLSEN, Captain, USAF; M.Ed., St. Thomas 
College, is a training officer in the Training Policy and 
Standards Division, Headquarters Air Training Command, 
Randolph Air Force Base, Texas. Before entering the Air 
Force in 1953, he taught instrumental music in the Minne- 
sota schools. He is Editor of the NSPI Journal and former 
President of the San Antonio Society for Programmed 
Instruction. 

ROBERT M. OPDYCKE, Captain, is now Chief of the 
Randolph Instructional Systems Development Team. He 
was Staff Training Methods Officer in the Directorate of 
the Assisant Chief of Staff for Training Development; 
Training Methods Division, Headquarters Air Training 
Command, Randolph AFB, Texas. He taught advanced 
flying at several Air Force locations. He has assisted in 
the writing of numerous training publications in the Air 
Training Command. 

JOHN PIERCE-JONES, Ph.D., University of Cali- 
fornia, Berkeley, California, is presently Professor of 
Educational Psychology and Director, Personnel Services 
Research Center at the University of Texas, Austin, Texas. 
He has taught at the University of Oregon and Pennsyl- 
vania State University. He has published numerous re- 
search articles in behavioral science journals and books. 
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DALE W. PODSHADLEY, D.D.S., San Francisco, Cal- 
ifornia, is Chief of the Education Research Program, Den- 
tal Health Center, Division of Dental Public Health and 
Resources, Public Health Service, U.S. Department of 
Health, Education and Welfare. He previously was with 
the Illinois Department of Welfare. He is the author of 
several papers on dental manpower resources and subjects 
related to public health practice. 

W. JAMES POPHAM, Ed.D., Los Angeles, California, 
is Assistant Professor of Education at the University of 
California, Los Angeles. He is the author of Understand- 
ing Educational Statistics (1964) and over 50 research 
articles and papers dealing with instruction and evaluation 
topics. He previously taught at Kansas State College of 
Pittsburg and San Francisco State College. 

ROBERT 7. PURSLEY, Major, USA, is assigned to 
the Medical Field Service School, Brooke Army Medical 
Center, Fort Sam Houston, Texas. He is the author of 
several articles appearing in professional journals describ- 
ing various methods of harnessing body motions for arti- 
ficial arms. Major Pursley received his Bachelor of Science 
Degree from the University of Maryland. 

GUSTAVE 7. RATH, Ph.D., Evanston, Illinois, is 
Associate Professor of Industrial Engineering and Manage- 
ment Science, Northwestern University. He conducted ex- 
perimental research on programming the IBM 650. He 
conducted programmer training for Raytheon Corporation 
and for USAF at Goodfellow AFB. Also principal investi- 
gator, on USAF contract for development of a pro- 
grammed course on Program and Evaluation Review 
Technique (PERTeach). 

EVELYN REVELS, M.S., Trinity University, San An- 
tonio, Texas, is Educational Secretary, Board of Nurse 
Examiners for the State of Texas. She was previously 
assistant director of the Robert B. Green School of Nurs- 
ing, San Antonio, Texas. 

RUSSELL E. ROBBINS, Ed.D., Dekalb, Illinois, is As- 
sistant Professor of Education, Northern Illinois Univer- 
sity. He has held teaching and administrative positions 
in the public schools of Illinois. From 1957 to 1959 he 
was Director of the International Petroleum Company 
Staff School, Peru, South America. 

GLENN H. ROGERS, B.A., M.A., University of Den- 
ver, is now Director of Learning Systems, a research cen- 
ter for accelerated learning techniques, Seattle, Washing- 
ton. He was formerly Assistant Professor of Speech and 
Director of Forensics, McMurry College, Abilene, Texas; 
Chairman of Speech and Drama, McCook College, Ne- 
braska; and taught at Purdue University and State Uni- 
versity of Iowa. Author of Aids in Teaching Language 
Arts (1963). 

WILLIAM F. RYAN, Ed.M., Castleton-on-Hudson, 
New York, is associate in Educational Communications 
of State Education Department, Albany, N. Y. He was 
previously assistant state audio-visual coordinator for the 



Commonwealth of Massachusetts, graduate instructor in 
audio-visual administration and education for the Massa- 
chusetts Division of University Extension, participant in 
NDEA foreign language studies and audio-visual coordi- 
nator and classical and foreign language studies at Chica- 
pee (Mass.) public schools. 

JERRY SHORT, Ph.D., University of Houston 1961, 
is a Research Scientist, Training and Education Program, 
American Institute for Research. He is also consultant in 
programmed instruction and training technology for the Air 
Training Command, USAF. 

ROBERT G. SMITH, JR., Ph.D., is representative of 
the Human Resources Research Office (HUMRRO) at 
Headquarters, U. S. Continental Army Command, Fort 
Monroe, Virginia. He has performed military psychologi- 
cal research for both the Air Force and Army, and taught 
at Texas Technological College. He is President of the 
National Society for Programmed Instruction (1963- 
1964). 

ARTHUR R. STEIGER, Major, USAF, is a Aeronaut- 
ical Engineer, USAF Academy, Director of Instructional 
Research and Associate Professor of Aeronautics until his 
reassignment in October 1963. He has a BSME, Purdue 
University and a M.S., Aeronautical Engineering, USAFIT. 

PASCHAL N. STRONG, Ph.D., is an associate pro- 
fessor of Psychology at Texas Tech. He is Vice President 
of INRAD; former editor of AID and is an Air Force 
Consultant on training. His research is concerned with 
basic learning, theory and physiological correlates of learn- 
ing. 

RUDOLPH C. TROIKE, Ph.D., is Assistant Professor 
of English for Foreign Students Program at the University 
of Texas, Austin, Texas. He was previously Chief, Mate- 
rials Preparation and Linguistic Research Branch, George- 
town University English Language Program, Ankara, Tur- 
key. He is the author of articles and papers on American 
Indian languages, ethnology, and machine translation. 

JOSEPH A. TUCKER, JR., Ph.D., Director of Train- 
ing Research, Cresap, McCormick & Paget, management 
consultants; has been active in programmed instruction 
since 1954. He worked as a research psychologist with 
the Air Force (1950-1957). A former high school teacher, 
he has taught at the University of Delaware, University of 
California, and Upsala College. 

HANNELORE F. VANDERSCHMIDT, Ed.M., Har- 
vard University, is Programming Coordinator at D. C. 
Heath & Company. Mrs. Vanderschmidt has taught at 
programmed instruction workshops at the University of 
Rochester and at the Denver public schools. She is co- 
translator of Karl Jaspers’ The Idea of the University 
(Beacon Press, 1959). 

M. S. WHITE, M.D., is Major General of the USAF 
and associated with the Federal Aviation Agency. U.S. 
Army-U.S. Air Force from 1931 to the present. Formerly 
Dr. White was Director of the Department of Aviation 
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Medicine, School of Aviation Medicine and Aeromedical 
Department, AAF School of Applied Tactics. He was 
command surgeon of the major commands of the USAF. 
Since 1959 he has been with the Air Training Command. 

BERNARD W. YAEGER, M.S., Purdue University, In- 
dustrial Psychology, is Senior Training Technologist at 
Honeywells California Ordnance Center, West Covina, 
California. His past work includes research in training and 
learning theory and experimental Human Factors engi- 
neering and teaching machine design. 

MICHAEL A. ZACCARIA, Ph.D., is Chief of the 
Training Evaluation Branch, Lackland AFB, San Antonio, 
Texas. He is also a lecturer in Education Psychology at 
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Our Lady of the Lake College, San Antom'o. He pre- 
viously taught at University of Houston, Texas, and at 
Trinity University, San Antonio. He is the author of many 
scientific and education papers on training technology, 
measurement, and training research. 

VIRGINIA ZACHERT, PhJD., is Research Associate in 
Medical Education in the Department of 01 stetrics & 
Gynecology of the Medical College of Georgia, Augusta, 
Ga. She previously directed the Air Force-US Industries 
Teaching Machine 3-year project at Keesler AFB, Missis- 
sippi. Prior to that she had been a researcher with indus- 
tries and the military services as consultant, contractor, 
civil servant and Air Force officer. She has written numer- 
ous reports and articles in her areas of interest. 



